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Executive  Summaiy 


“Are  we  winning?”  That  simple  question  has  proven  decidedly  difficult  for  the  U.S. 
military  to  answer  at  various  times  in  its  history,  and  in  recent  decades  has  given  rise 
to  formal  processes  for  trying  to  measure  progress  in  warfare — a  practice  that  has 
become  known  as  “operations  assessment.”  Over  the  past  decade,  CNA  analysts  have 
found  themselves  increasingly  involved  in  operations  assessment,  as  has  the  broader 
military  operations  research  community.  Yet,  to  our  knowledge,  there  is  no 
comprehensive  account  of  the  history  of  operations  assessment.  This  paper  aims  to 
serve  as  a  primer  on  the  subject  to  provide  context  on  how  operations  assessment 
has  grown  and  developed  over  the  past  several  decades.  We  hope  that  it  serves  as  a 
foundation  and  source  of  encouragement  for  a  more  robust  history  on  this  topic. 

Although  the  genesis  of  what  we  now  refer  to  as  operations  assessment  began  in 
World  War  11  with  the  growth  of  operations  research,  the  Vietnam  era  saw  the 
emergence  and  practice  of  the  concept  take  on  a  significance  that  was 
unprecedented.  As  the  United  States  became  immersed  in  a  counterinsurgency 
fraught  with  ambiguities,  nascent  operations  research  and  systems  analysis  (ORSA) 
analysts,  among  others,  strove  to  find  ways  to  measure  progress  on  the  ground.  The 
main  driving  force  behind  this  effort  was  Secretary  of  Defense  Robert  McNamara.  A 
strong  proponent  of  systems  analysis  and  other  quantitative  methods,  McNamara 
drove  the  idea  that  data  collected  on  the  ground  could  be  used  to  develop  accurate 
and  precise  measures  of  progress. 

McNamara’s  approach  took  the  form  of  three  primary  assessment  models:  body 
counts,  the  Hamlet  Evaluation  System,  and  the  Southeast  Asia  Analysis  Reports.  Each 
depended  on  highly  quantitative  inputs  to  produce  measurements  that  reflected  the 
degree  to  which  the  U.S.  military  was  meeting  its  objectives  in  Vietnam.  While  this 
effort  represented  a  highly  innovative  approach  to  a  very  difficult  problem,  the 
process  proved  highly  unreliable  as  an  accurate  reflection  of  progress.  The  emphasis 
on  quantitative  measures  obviated  the  inclusion  of  qualitative  judgment  and  ignored 
the  disparate  nature  of  the  environment  in  Vietnam,  failing  to  account  for  nuance 
and  context.  By  the  end  of  the  war,  policymakers  and  Defense  leaders  were  wholly 
disenchanted  with  the  assessments  process  and  summarily  dismissed  it  as  invalid 
and  useless. 

In  the  years  following  Vietnam  and  leading  up  to  the  end  of  the  Cold  War,  doctrine 
and  thinking  on  operations  assessment  was  minimal  and  tended  to  revert  back  to 
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pre-Vietnam  methods  like  battle  damage  assessment  and  the  commander’s  estimate. 
The  explanation  for  this  lack  of  emphasis  lies  primarily  in  the  geopolitical  dynamics 
that  characterized  the  Cold  War.  Because  the  U.S.  defense  posture  was  oriented 
almost  exclusively  toward  countering  the  Soviet  Union,  it  became  much  more 
important  to  compare  and  measure  the  capabilities  of  the  two  nations’  militaries, 
rather  than  the  effectiveness  of  their  operations.  Since  the  two  superpowers  never 
actually  engaged  in  “hot”  conflict,  the  perceived  need  for  a  robust  operations 
assessment  process  was  virtually  nonexistent.  Coupled  with  the  backlash  against  the 
highly  quantitative  Vietnam-era  approach,  the  later  years  of  the  Cold  War  saw 
operations  assessment  fall  into  relative  obscurity. 

With  the  collapse  of  the  Soviet  Union,  the  United  States  took  on  a  new  role  as  the  sole 
superpower,  and  with  that  shift  came  new  global  responsibilities.  U.S.  engagement 
around  the  world  began  to  grow  and  the  need  for  an  assessments  process  re- 
emerged.  At  the  same  time,  the  world  was  entering  the  Information  Age,  where  vastly 
improved  computing  capabilities  were  prompting  sweeping  changes  in  every  sector, 
including  defense. 

With  these  changes  came  several  new  conceptual  undercurrents  that  would 
ultimately  influence  the  development  of  a  new  approach  to  operations  assessment. 
Network-centric  warfare,  the  “Revolution  in  Military  Affairs,”  and,  ultimately,  effects- 
based  operations  all  grew  out  of  the  idea  that  advanced  technologies  would  usher  in 
a  new  era  in  warfare— one  that  made  war  almost,  if  not  entirely,  calculable.  For 
operations  assessment,  these  changes  signaled  a  similar  shift  in  thinking,  back  to 
more  quantitative  approaches.  The  emergence  of  effects-based  assessment,  which 
emphasized  the  use  of  measures  of  performance  and  measures  of  effectiveness- 
expressed  quantitatively— to  determine  the  level  of  progress,  became  the  dominating 
framework  for  operations  assessment. 

The  wars  in  Iraq  and  Afghanistan  have  brought  operations  assessment  significance 
and  attention  unparalleled  in  U.S.  military  history.  Initially  relying  on  effects-based 
assessment  to  provide  the  necessary  framework,  assessors  in  both  conflicts  met  with 
hindrance  after  hindrance  as  the  shortcomings  of  this  approach  became  more 
apparent.  The  discormect  between  counterinsurgency  theory  and  the  assessments 
process  that  had  plagued  operations  assessment  in  Vietnam  re-emerged  and  the 
result  has  been  equally  frustrating.  The  promise  of  technological  advancement  and 
the  effects-based  framework  to  help  make  sense  of  the  vast  amount  of  data  coming 
from  both  theaters  has  fallen  short.  Once  again,  the  failure  of  the  process  to  account 
for  local  context  and  the  pitfalls  in  trying  to  quantify  complex  dynamics  has  made 
the  production  of  accurate  and  useful  assessments  a  persistently  elusive  aim. 

This  brief  history  of  operations  assessment  has  revealed  important  trends  and 
provides  insight  into  how  the  process  can  be  improved  in  the  future.  The  oscillation 
between  quantitative  and  qualitative  inputs  and  approaches  to  assessment  has  been 
a  persistent  theme.  Also,  the  emphasis  on  operations  assessments  seems  to  ebb  and 
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flow  according  to  the  level  of  demand  for  them,  especially  from  high-level  Defense 
Department  leadership.  The  level  of  demand,  in  turn,  seems  proportional  to  a  lack  of 
obvious  progress.  When  progress  is  difficult  to  demonstrate  and  communicate,  which 
is  especially  true  in  counterinsurgency,  there  is  a  greater  demand  for  accurate 
assessments.  Conversely,  in  eras  like  the  Cold  War,  where  the  focus  resided  in  pre¬ 
conflict  comparisons  of  forces,  demand  for  operations  assessment  was  almost 
nonexistent. 

Recent  trends  in  operations  assessment  include  some  high-level  commanders’  use  of 
assessments  to  help  bolster  the  message  they  wish  to  communicate  to  higher 
leadership  and  public  audiences.  In  an  effort  to  gain  the  support  of  policymakers  and 
other  civilian  audiences,  some  commanders  have  selected  metrics  and  molded 
assessments  in  ways  that  paint  a  picture  of  progress  consistent  with  their  intended 
message.  This  development  has  had  significant  implications  for  operations 
assessment,  and  the  processes  at  numerous  high-level  commands  have  undergone 
substantial  overhauls  as  subsequent  commanders  seek  an  appropriate  balance  in 
satisfying  the  high-level  civilian  audience  while  building  assessments  that  still 
provide  meaningful  inputs  to  their  commands’  internal  plarming  processes. 

The  recent  high  level  of  demand  for  assessments  has  brought  about  what  General 
James  Mattis  referred  to  as  a  “renaissance”  in  operations  assessment.  Currently  there 
is  great  momentum  behind  efforts  to  build  a  workable  and  effective  framework  for 
assessments,  which  has  given  rise  to  revisions  of  major  U.S.  doctrinal  publications  on 
the  subject.  There  is  great  opportunity  at  this  time  to  take  lessons  learned  from 
operations  assessment  history  and  apply  them  to  the  development  of  processes  that 
avoid  the  problems  of  the  past  and  anticipate  future  considerations. 

Challenges  such  as  “big  data”  and  limited  analytic  capacity,  increased  emphasis  on 
operations  assessment  amidst  discord  as  to  how  to  conduct  it,  demands  of 
competing  audiences,  lack  of  training  and  education  for  practitioners,  and  the 
uncertainties  and  ambiguities  of  warfare  will  continue  to  make  the  conduct  of 
operations  assessment  a  challenging  endeavor.  Yet,  if  we  are  to  avoid  adding  another 
chapter  of  failing  and  frustration  to  the  history  of  operations  assessment,  it  is 
imperative  for  the  U.S.  military  to  seize  the  momentum  and  opportunity  that 
currently  exists  to  re-examine  and  redesign  effective  means  of  assessing  progress  in 
future  operations. 
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In  more  than  a  decade  of  war  in  Iraq  and  Afghanistan,  the  United  States’  armed 
forces  have  encountered  myriad  challenges  in  their  efforts  to  execute  simultaneous 
counterinsurgency  campaigns.  Assessing  progress  on  the  ground— what  we  will  refer 
to  here  as  “operations  assessment” — has  proven  an  especially  challenging  task.*  For 
CNA  analysts,  as  well  as  uniformed  operations  research  and  systems  analysis  (ORSA) 
personnel,  the  operations  assessment  process  has  grown  in  importance  and 
consumed  increasing  analytic  resources  over  the  course  of  Operation  Iraqi  Freedom 
(OIF)  and  Operation  Enduring  Freedom  (OFF).  While  there  have  been  new  and 
Innovative  approaches  to  operations  assessment  in  recent  years,  there  has  also  been 
a  great  deal  of  relearning  old  lessons  from  previous  eras  (under  different  conditions). 
This  relearning  is,  at  least  in  part,  attributable  to  the  lack  of  a  comprehensive  history 
of  operations  assessment. 

Examining  the  history  of  assessment  can  help  practitioners  be  successful  and  avoid 
relearning  past  lessons,  and  it  can  better  prepare  the  next  generation  of  practitioners 
for  ascertaining  progress  in  future  operations.  This  paper  has  two  aims:  First,  to 
serve  as  a  primer  on  operations  assessment  by  laying  out  some  of  the  major  trends 
tn  its  development  and  employment  over  time.  Second,  to  serve  as  a  foundation  and 
source  of  encouragement  for  the  research  and  production  of  a  more  complete  history 
of  operations  assessment. 

The  process  that  we  refer  to  today  as  operations  assessment  finds  its  origins  in  the 
World  War  II  era.  During  that  time,  U.S.  armed  forces  began  to  dedicate  significant 
resources  to  the  relatively  new  field  of  operations  research  (OR).^  Current  U.S.  joint 


1.  The  U.S.  Joint  Staff,  Commander’s  Handbook  for  Assessment  Planning  and  Execution 
(Washington,  DC:  U.S.  Joint  Staff,  2011)  describes  the  activity  of  assessment  (referred  to  in  this 
document  as  operations  assessment)  as  follows:  Commanders,  assisted  by  their  staffs  and 
subordinate  commanders,  along  with  interagency  and  multinational  partners  and  other 
stakeholders,  will  continuously  assess  the  operational  environment  and  the  progress  of  the 
operation  toward  the  desired  end  state  in  the  time  frame  desired.  Based  on  their  assessment, 
commanders  direct  adjustments,  thus  ensuring  the  operation  remains  focused  on 
accomplishing  the  mission.  Assessment  is  appUcable  across  the  range  of  military  operations.  It 
offers  perspective  and  insight,  and  provides  the  opportunity  for  self-correction,  adaptation, 
and  thoughtful  results-oriented  learning,  (p.  vn) 

2.  The  history  of  operations  research  is  weU-documented,  while  the  purpose  here  is  to  examine 
the  history  of  operations  assessment.  See  Maurice  W.  Kirby,  Operational  Research  in  War  and 
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doctrine  defines  OR  as  “the  analytical  study  of  military  problems  undertaken  to 
provide  responsible  commanders  and  staff  agencies  with  a  scientific  basis  for 
decision  on  action  to  improve  military  operations.”^  With  the  adoption  of  OR  among 
the  armed  forces,  statistical  models  and  quantitative  measures  became  common  in 
the  conduct  of  warfare,  bringing  much  more  “science”  to  traditional  ideas  of  the  “art” 
of  war.  Since  that  time,  the  development  of  OR  has  brought  substantial  changes  to 
the  way  the  United  States  and  its  partners  fight  their  wars.  Much  of  the  thinking 
behind  contemporary  operations  assessment  is  founded  in  OR  principles, 
emphasizing  quantitative  metrics  to  demonstrate  progress.  However,  practitioners  of 
these  methods  have  also  encountered  significant  obstacles,  often  resulting  in  the 
movement  away  from  the  quantitative  in  favor  of  more  balanced  approaches. 

As  we  will  see,  the  major  trends  in  operations  assessment  essentially  follow  the  path 
of  a  pendulum  swinging  between  highly  quantitative  methods  and  loose  guidelines 
that  place  much  of  the  burden  of  assessment  on  the  commander’s  judgment  and 
staff  inputs.  It  has  also  become  clear  through  this  study  that  the  emphasis  placed  by 
the  military  on  operations  assessment  is  typically  proportional  to  the  level  of 
demand  for  assessments  by  policymakers  and  the  U.S.  public.  There  have  been 
prolonged  periods  in  U.S.  history  where  the  absence  of  open  conflict  has  obviated  the 
perceived  need  for  operations  assessment,  relegating  the  practice  to  a  comparatively 
minor  role  in  the  plarming  process.  This  oscillation  over  time  is  the  result  of  many 
factors,  including  the  level  of  U.S.  involvement  in  armed  conflict,  the  global  balance 
of  power,  technological  advances,  and  economic  and  business  practices.  The 
remainder  of  this  paper  will  trace  the  development  of  operations  assessment  in  light 
of  these  factors.  We  will  begin  in  the  Vietnam  era,  follow  the  decline  of  assessment 
during  the  Cold  War  and  a  renewed  emphasis  that  accompanied  the  Information  Age, 
and  examine  the  struggles  and  evolution  of  assessment  during  the  wars  in  Iraq  and 
Afghanistan.  Having  examined  this  history,  we  will  conclude  with  some  thoughts  on 
what  the  future  might  hold  for  operations  assessment. 


Peace:  The  British  Experience  from  the  1930s  to  1970  (London;  Imperial  College  Press,  2003). 
See  also  Charles  R.  Shrader,  History  of  Operations  Research  in  the  United  States  Army,  3  vols., 
vol.  1  (Washington,  DC,  2006). 

3.  U.S.  Joint  Staff,  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms,  Joint 
PubUcation  1-02  (Washington,  DC:  U.S.  Joint  Staff,  2014),  p.  197. 
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The  United  States  emerged  from  World  War  II  a  changed  nation  in  a  changed  world. 
Old  power  balances  had  crumbled,  and  the  United  States  was  moving  forward  into 
unfamiliar  territory  as  a  world  superpower  in  a  volatile  security  environment.  To 
stem  the  spreading  influence  of  the  other  superpower — the  Soviet  Union — U.S. 
leaders  adopted  the  policy  of  Containment,  which  sought  to  stem  the  spread  of 
Soviet  influence  without  engaging  in  a  direct  confrontation  with  the  USSR.  Military 
and  foreign  policy  budgets  swelled,  and  the  preponderance  of  U.S.  defense  and 
diplomatic  resources  were  dedicated  to  keeping  Soviet  power  contained.  This  policy 
had  far-reaching  implications,  none  more  costly  than  the  Vietnam  War.  In  Vietnam, 
U.S.  armed  forces  encountered  a  very  different  kind  of  warfare  from  previous  large- 
scale  conflicts,  and  the  ambiguities  associated  with  fighting  a  counterinsurgency 
brought  operations  assessment  to  the  forefront  for  the  first  time  in  U.S.  military 
history. 


Ihe  ''McNamam  Revolution" 


The  Vietnam  War  was  a  revolutionary  period  in  the  development  of  operations 
assessment.  It  represented  the  first  effort  to  build  a  systematic  assessments  process 
to  measure  progress  in  a  counterinsurgency.  The  methods  employed  tended  to  be 
highly  quantitative,  reflecting  the  scientific  undercurrents  of  the  time,  including 
operations  research  and  its  companion  field  of  systems  analysis.  One  of  the  most 
influential  advocates  of  the  systems  analysis  approach  was  Secretary  of  Defense 
Robert  McNamara.  His  experience  at  Harvard  Business  School,  in  the  World  War  II 
Army  Air  Forces  Statistical  Control  Division,  and  subsequently  at  Ford  Motor 
Company,  formed  the  foundation  on  which  he  built  the  ideas  he  brought  to  the 
Department  of  Defense. 

McNamara’s  approach  was  data-centered  and  emphasized  mathematical  precision. 
He  had  found  success  at  Ford,  where  his  methods  helped  generate  a  remarkable 
financial  recovery  of  a  business  that  was  failing  when  he  joined  it.  When  he  was 
tapped  to  be  Defense  Secretary  in  1961,  the  Pentagon  was  beset  by  bureaucratic 
inefficiency,  and  Defense  leaders  were  increasingly  occupied  with  the  growing  U.S. 
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commitment  in  Vietnam.  McNamara  was  intent  on  applying  his  methods  to  both 
problems.^ 

Prior  to  his  arrival  in  Washington,  McNamara  spent  15  years  at  Ford,  eventually 
taking  over  as  president  in  1960.  McNamara,  the  “high  priest  of  the  new  religion  of 
rational,  scientific  decision  making,”^  came  to  the  Pentagon  intent  on  streamlining 
and  improving  operating  procedures  as  he  had  streamlined  and  improved  production 
at  Ford.  Dubbed  the  “McNamara  Revolution,”  his  time  in  office  produced  lasting 
changes,  including  the  emergence  of  operations  research  and  systems  analysis 
(ORSA).®  Operations  research  was  already  decades  into  its  development  by  1961,  but 
under  McNamara  it  was  deliberately  paired  with  systems  analysis.  The  growth  of 
ORSA  certainly  made  waves  in  the  Pentagon  bureaucracy,  but  it  also  spilled  over  into 
the  conduct  and  assessment  of  the  Vietnam  War.  Since  McNamara’s 
institutionalization  of  the  practice,  ORSA  analysts  have  typically  constituted  some,  if 
not  most,  of  the  personnel  responsible  for  operations  assessment. 


Assessing  PiDgiess  in  Vietnam 


When  Robert  McNamara  took  over  at  the  Pentagon  under  newly  elected  President 
John  F.  Kennedy,  the  U.S.  commitment  in  Vietnam  was  relatively  small,  consisting  of 
military  advisors.  Kennedy  and  McNamara  increased  the  number  of  advisors 
significantly,  and  under  President  Johnson  the  U.S.  role  escalated  from  a  robust 
advisory  effort  to  full-scale  armed  conflict  by  1965.  The  irregular  nature  of  the  war 
posed  formidable  challenges  for  U.S.  forces  in  every  aspect,  including  assessments. 

As  the  war  progressed  and  the  United  States  pledged  more  resources  to  it,  McNamara 
and  his  staff  devoted  significant  effort  to  developing  methods  to  measure  progress 
in  a  conflict  plagued  by  ambiguities  and  inconsistencies.  They  relied  on  the  same 


4.  This  paper  provides  a  cursory  overview  of  the  assessments  process  in  Vietnam.  For  a 
detailed  examination  of  the  subject,  see  Ben  Coimable,  "Embracing  the  Fog  of  War:  Assessment 
and  Metrics  in  Counterinsurgency,"  (2012).  See  also  Gregory  A.  Daddis,  No  Sure  Victory: 
Measuring  U.S.  Army  Effectiveness  and  Progress  in  the  Vietnam  War  (New  York,  NY:  Oxford 
University  Press,  2011). 

5.  Charles  R.  Shrader,  History  of  Operations  Research  in  the  United  States  Army,  3  vols.,  vol.  2 
(Washington,  DC,  2008),  p.  40. 

6.  While  the  two  are  not  wholly  distinct  from  each  other,  operations  research  tends  to  focus  on 
how  to  make  a  given  system  operate  effectively,  while  systems  analysis  is  concerned  with  the 
bigger  picture  of  whether  that  system  is  right  for  the  job  and  how  it  fits  with  other  systems  in 
the  larger  effort  toward  achieving  an  objective;  see  ibid.,  pp.  55-56.  While  it  is  important  to 
note  that  ORSA  should  not  be  equated  with  operations  assessment,  it  did  contribute  greatly  to 
the  foundation  of  scientific  methods  on  which  those  assessments  rested. 
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types  of  quantitative,  systems-based  methods  that  McNamara  had  applied  at  the 
Pentagon  to  help  make  sense  of  what  was  happening  on  the  ground  in  Vietnam.  A 
likeminded  official  in  this  effort  was  the  head  of  the  Southeast  Asia  Division  in  the 
newly  created  Office  of  Systems  Analysis  (OSA)  at  the  Pentagon,  Thomas  Thayer.^ 
Endeavoring  to  bring  precise  metrics  of  progress  to  the  conflict,  McNamara,  Thayer, 
and  the  so-called  “Whiz  Kids”  of  the  Pentagon  built  an  assessments  process 
essentially  from  scratch.®  As  with  many  innovations,  their  approach  generated 
pushback  from  the  start.  Senior  military  officials  in  particular  balked  at  what  seemed 
a  categorical  dismissal  of  their  own  experience  and  judgment  in  assessing 
operations.®  The  backlash  against  these  new  methods  is  particularly  evident  in  the 
years  after  Vietnam  (discussed  in  the  next  section). 

The  Pentagon’s  penchant  for  hard  numbers  was  not  new,^”  but  for  quantitative 
metrics  to  be  useful,  they  must  be  based  on  a  sound  strategic  model.  In  1966,  the 
U.S.  adopted  a  strategic  model  centered  on  attrition  of  Viet  Cong  fighters.  The 
corresponding  attrition  model  for  assessment  was  manifested  most  notably  in  the 
infamous  “body  count”  metric."  Body  counts  were  part  of  the  monthly  reports  that 
the  Military  Assistance  Command,  Vietnam  (MACV)  submitted  detailing  progress  on 
the  ground  throughout  the  country.  These  periodic  reports  evolved  over  time  as  the 
assessments  process  changed,  but  the  body  count  metric  is  of  particular  note 
because  it  represented  the  primary  measure  of  progress  from  1966  to  1968. 

The  goal  for  using  the  body  count  metric  was  to  quantitatively  identify  a  desired 
“crossover  point”  where  attrition  of  enemy  forces  occurred  faster  than  the  enemy 
could  replenish  them."  Units  across  Vietnam  would  submit  numbers  of  enemy  killed, 
resulting  in  a  substantial  amount  of  raw  data  that  analysts  would  process  (with  the 
aid  of  new  computing  technologies)  at  MACV  headquarters.  However,  many  body 
count  estimations  were  highly  inaccurate,  either  due  to  an  inability  to  confirm  the 
number  of  enemy  killed  or  to  a  deliberate  inflation  of  the  numbers  in  an  effort  to 


7.  Connable,  "Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency,"  p.  99. 

8.  The  “Whiz  Kids”  were  relatively  young,  intelligent,  aggressive  minds  brought  on  for  their 
knowledge  of  systems  analysis  and  other  scientific  approaches.  Their  approach  tended  to 
alienate  senior  mihtary  leaders,  however,  who  considered  them  arrogant  and  ignorant  of  how 
warfare  actuaUy  works.  For  more  on  the  “Whiz  Kids,”  see  Shrader,  History  of  Operations 
Research  in  the  United  States  Army,  voL  1. 

9.  Ibid.,  p.  63-64. 

10.  World  War  II  saw  the  advent  of  an  increasingly  robust  operations  research  capability,  which 
relied  on  quantitative  and  scientific  methods  to  understand  and  improve  friendly  force 
capacity  and  capability.  U.S.  forces  used  similar  methods  in  the  Korean  War. 

11.  Daddis,  No  Sure  Victory:  Measuring  U.S.  Army  Effectiveness  and  Progress  in  the  Vietnam 
War,  p.  91. 

12.  Ibid. 
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please  higher  headquarters.  Top-down  demand  for  progress  toward  the  crossover 
point  drove  units  to  misrepresent  their  numbers  to  satisfy  their  leadership. 
Inconsistencies  among  units  in  how  and  whom  they  counted  (for  example,  some 
units’  numbers  included  civilian  casualties)  further  added  to  the  dubiousness  of  the 
metric.*^  However,  regardless  of  accuracy,  the  most  fundamental  problem  with  body 
counts  was  that  they  were  highly  unsuitable  as  a  measure  of  effectiveness  or 
progress  in  a  counterinsurgency— in  other  words,  they  were  a  metric  derived  from  a 
failed  strategic  model.  As  we  will  see  in  a  later  section,  the  search  for  the  “right” 
metrics  for  failing  strategic  models  was  a  problem  that  recurred  in  both  Iraq  and 
Afghanistan. 

Another  notable  assessment  tool  used  to  inform  decision  makers  in  Vietnam  was  the 
Hamlet  Evaluation  System  (HES).  The  HES  was  designed  with  the  intent  to  provide 
senior  decision  makers  with  an  objective  account  of  progress  on  tbe  ground.  The 
system  was  based  on  survey  data  collected  monthly  at  the  sub-village,  or  “hamlet,” 
level.  Since  there  were  11,000-13,000  hamlets  surveyed  each  month,  the  sheer 
volume  of  data  propelled  the  system  toward  its  first  major  flaw:  to  manage  the  data, 
results  were  distilled  to  numeric  scores.^'*  To  produce  those  scores,  HES  used  a  set  of 
six  major  factors,  each  of  which  had  an  associated  indicator.  The  output  from  the 
surveys  was  a  set  of  numbers  that  could  be  highly  arbitrary,  with  no  mechanism  to 
account  for  qualitative  nuance.  The  numbers  were  then  aggregated  to  create  a  single 
“overall  pacification  number”  that  was  reported  to  the  highest  levels  of  Defense 
leadership. This  aggregation  points  to  the  second  major  problem  with  HES,  which 
was  the  highly  centralized  nature  of  data  management  and  analysis.  In  Vietnam, 
where  the  number  of  boots  on  the  ground  translated  into  a  proportionately  large 
capacity  to  collect  information,  analysts  had  to  turn  reams  of  data  from  highly 
disparate  environments  into  manageable  assessments.  Unfortunately,  tbis  effort 
precluded  tbe  inclusion  of  nuanced  factors  peculiar  to  local  hamlets  and  villages, 
resulting  in  the  loss  of  critical  information  and  painting  a  distorted  picture  of 
progress. 

Not  all  assessments  in  Vietnam  lacked  qualitative  injects,  however.  In  his  role  at  OSA, 
Thomas  Thayer  developed  an  influential  operations  assessment  product  that  used 
body  counts,  HES  reports,  and  other  data  as  its  inputs.  His  bimonthly  “Southeast  Asia 
Analysis  Reports”  applied  rigorous  and  systematic  analytic  methods  to  vast  amounts 
of  quantitative  and  qualitative  data  from  the  field.  The  approach  was  holistic  and 
comprehensive,  focused  on  identifying  patterns  and  trends  that  would  be  useful  in 

13.  Connable,  "Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency,"  pp. 
108-10. 

14.  Ibid.,  p.  114. 

15.  Ibid.,  p.  119;  Erwin  R.  Brigham,  "Pacification  Measurement  m  Vietnam:  The  Hamlet 
Evaluation  System,"  m  SEATO  Internal  Security  Seminar  (Manilal968),  p.  2. 
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shaping  U.S.  strategy.  Thayer’s  reflections  in  his  book,  War  Without  Fronts,  are  honest 
about  tbe  shortcomings  of  the  process.  Reflecting  on  the  pitfalls  of  the  body  count 
metric,  for  example,  he  writes  that  “the  adoption  of  an  attrition  strategy  by  the 
United  States  (‘destroy  the  enemy  forces  faster  than  they  can  be  replaced’) 
complicated  the  process  by  making  the  number  of  communist  forces  a  prime 
measure  of  success  or  failure. However,  Tbayer  most  often  highlights  deficiencies 
in  the  data  and  the  analysis,  rather  than  the  process,  in  his  review. He  holds  firmly 
to  the  idea  that  systems  analysis  and  its  associated  methods  could  provide  a 
workable  framework  for  assessments  in  counterinsurgency. 


Implic  ations  for  Opeiations  Assessment 


The  Vietnam  era  was  a  transformative  period  in  U.S.  history  for  operations 
assessment.  The  sheer  amount  of  time  and  resources  dedicated  to  the  effort  was 
unprecedented  to  that  point.  Even  World  War  II — a  conflict  conducted  on  a  massive 
scale— did  not  place  such  a  heavy  emphasis  on  collecting  data  and  developing 
methods  to  measure  progress.^**  Likewise,  the  application  of  systems  analysis  to 
develop  a  rigorous  assessments  process  was  an  innovation  of  marked  significance. 
The  foundations  had  been  laid  decades  earlier  with  the  advent  of  operations 
research,  but  Vietnam  saw  the  implementation  of  those  fundamental  ideas  on  a  large 
scale.  This  period  also  marked  the  first  time  U.S.  military  leaders  attempted  to  build 
a  workable  framework  for  assessments  in  a  counterinsurgency.  Much  of  what  exists 
today  on  that  topic  grew  out  of  the  American  experience  in  Vietnam. 

Historical  treatment  of  these  developments  in  operations  assessment  tends  to  be 
highly  critical.  While  the  U.S.  strategy  and  approach  to  counterinsurgency  in  Vietnam 
was  highly  ineffective,  flawed  assessments  undoubtedly  contributed  to  the  ultimate 
failure  of  U.S.  efforts  there.  As  one  author  writes  when  considering  McNamara’s 
legacy,  the  “single-minded  emphasis  on  rational  analysis  based  on  quantifiable  data” 


16.  Thomas  C.  Thayer,  War  without  Fronts:  The  American  Experience  in  Vietnam  (Boulder,  CO: 
Westview  Press,  1985),  p.  28. 

17.  Ibid.,  p.  5. 

18.  The  U.S.  military  largely  created  their  operations  assessment  process  “on  the  fly.”  WMe 
Vietnam  was  the  first  major  counterinsurgency  in  the  American  military  experience,  other 
nations  had  engaged  in  similar  conflicts  (e.g.,  the  British  experience  with  the  Malayan 
Emergency).  However,  U.S.  military  leaders  did  not  draw  on  lessons  learned  from  those  efforts 
in  building  their  own  assessments  process,  but  Instead  started  essentially  from  scratch  (see 
Daddis,  No  Sure  Victory:  Measuring  U.S.  Army  Effectiveness  and  Progress  in  the  Vietnam  War,  p. 
12). 
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that  he  advocated  “led  to  grave  errors.”^®  The  translation  of  demonstrably  effective 
methods  in  improving  the  business  side  of  Defense  (e.g.,  weapons  development, 
acquisitions,  and  inventory  control)  to  the  assessments  process  proved  untenable.^” 
Despite  the  efforts  of  those  like  Thayer,  who  sought  to  account  for  qualitative  factors 
in  his  reports,  numbers  and  statistics  dominated  the  assessment  approach,  glossing 
over  highly  relevant  complexities  that  characterize  counterinsurgency  operations. 
Not  only  were  the  results  of  the  assessments  process  unreliable,  but  there  was 
notable  disproportionality  between  the  resources  dedicated  to  operations 
assessments  and  their  utility.  Feeding  the  assessments  process,  as  Cormable 
describes,  “became  a  military  objective  unto  itself.”^^  Pressure  from  U.S.  policymakers 
to  produce  accurate  assessments  only  compounded  this  problem.  However,  absent  a 
sound  strategic  model  and  without  reliable  data  at  its  core,  the  assessments  process 
was  bound  to  produce  misleading  results. 

The  general  distaste  for  Vietnam  era  assessments  was  most  prevalent  in  the 
immediate  postwar  years,  when  Defense  leadership  resoundingly  dismissed 
McNamara  and  his  colleagues’  methods  as  ineffective  and  counterproductive. 
However,  their  application  of  systems  analysis  established  a  precedent  that  future 
leaders  would  revive.  Many  of  the  issues  that  plagued  Vietnam  assessments  have  re- 
emerged  in  the  counterinsurgency  environments  of  Iraq  and  Afghanistan.  Like 
Thayer,  many  of  the  more  recent  practitioners  of,  and  commentators  on,  operations 
assessment  have  asserted  that  the  process  is  valid,  if  only  we  select  a  sound  strategic 
model  and  the  associated  “right”  metrics.  However,  as  we  will  discuss  in  more  detail 
later  in  this  paper,  there  has  also  been  a  growing  chorus  of  voices  against  the  process 
in  recent  years. 


19.  Phil  Rosenzweig,  "McNamara  and  the  Evolution  of  Modern  Business  Management,"  Harvard 
Business  Review  (December  2010):  p.  92. 

20.  Shrader,  History  of  Operations  Research  in  the  United  States  Army,  vol.  2,  p.  66. 

21.  Connable,  "Embracing  the  Fog  of  War:  Assessment  and  Metrics  hr  Counterinsurgency,"  p. 
112. 
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Ihe  Cold  War 


As  the  U.S.  Department  of  Defense  limped  forward  after  the  grueling  experience  of 
Vietnam,  it  concentrated  its  (much  diminished)  resources  hack  on  the  Soviet  Union  as 
the  ultimate  threat  that  defined  and  shaped  Defense  policy  and  doctrine.  The  bipolar 
balance  of  power  between  the  two  countries  and  their  allies  fostered  a  constant 
tendency  to  compare  capabilities  and  view  policy  and  defense  considerations  in  light 
of  that  evaluation.  This  shift  had  implications  for  operations  assessment,  which  fell 
into  relative  obscurity  during  the  height  of  the  Cold  War. 


Ihe  Rise  of  Net  Assessment 


The  conclusion  of  U.S.  operations  in  Vietnam  brought  about  a  time  of 
reconsideration  in  U.S.  defense  policy.  Much  of  the  thinking  reflected  a  reaction 
agamst  Vietnam-era  approaches,  since  they  were  often  associated  with  the  failures 
the  U.S.  encountered  in  that  conflict.  One  of  the  major  conceptual  developments  that 
came  to  prominence  during  this  time  was  net  assessment.^^  One  author,  writing  in 
1990,  defines  net  assessment  as  “the  craft  and  discipline  of  analyzing  military 
balances. it  takes  a  strategic,  long-term  look  at  how  countries  measure  up  against 
one  another.  Such  a  comparative  approach  bears  some  similarities  to  the  correlation 
of  forces  doctrine  that  is  addressed  in  more  detail  below.  The  use  of  the  word  “net” 
refers  to  the  comparison  of  many  factors  (especially  friendly  and  enemy) 
simultaneously  and  drawing  analysis  from  the  “net  result”  of  those  comparisons.  Net 
assessment  also  reflects  the  post-Vietnam  rejection  of  systems  analysis  in  favor  of  a 
more  balanced  approach  that  emphasizes  the  importance  and  relevance  of 
qualitative  factors.  Net  assessment  embraces  complexity,  rather  than  seeking  to 
reduce  a  problem  to  simple,  quantitative  components,  and  emphasizes  human 


22.  The  Office  of  Net  Assessment  at  the  Pentagon  was  estabhshed  in  1973  to  provide  long- 
range  assessments  comparing  U.S.  strengths  and  capabilities  against  those  of  its  adversaries 
(i.e.,  the  Soviet  Union  and  its  allies). 

23.  Ehot  A.  Cohen,  Net  Assessment:  An  American  Approach,  JCSS  Memorandum  no.  29  (April 
1990),  p.  4. 
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thought  over  scientific  models  to  manage  that  complexity. Numbers  are  an 
important  input  to  net  assessment,  hut  they  are  taken  into  consideration  along  with 
many  other  factors  and  subjected  to  a  number  of  different  analytic  methods. 

Despite  these  notable  divergences  from  systems  analysis,  there  are  certain  practices 
common  to  both  approaches,  including  the  call  to  assess  military  effectiveness.^^ 
However,  in  the  absence  of  armed  conflict,  the  approaches  measure  the  anticipated 
effectiveness  of  friendly  forces,  rather  than  the  real-time  effectiveness  of  military 
operations.  The  1990  Joint  Military  Net  Assessment  (JMNA)^®  reflects  the  influence  of 
net  assessment  at  the  strategic  and  policy  level.  Its  lists  objectives  pertaining  to  “the 
potential  effectiveness  of  the  FY  1990  U.S.  force  ...  against  the  FY  1990  force  of 
potential  adversaries,”  as  well  as  “the  expected  effectiveness  of  anticipated  U.S. 
future  field  forces. As  we  will  show,  this  appraisal  of  one  force  against  another 
dominated  thinking  on  assessments  in  the  Cold  War.  Operations  assessment  was 
largely  irrelevant  during  this  period — and  is  not  explicitly  addressed  in  net 
assessment— since  it  was  deemed  necessary  only  once  open  conflict  begins. 


Development  of  U.S  Doctrine 


In  the  years  immediately  following  withdrawal  from  Vietnam,  the  Pentagon’s 
comparatively  meager  means  and  pessimistic  outlook  compelled  leaders  to  adopt  a 
highly  defensive  posture,  officially  labeled  “Active  Defense.”  Active  Defense  focused 
on  defending  Europe  against  potential  Soviet  encroachment  as  the  primary  objective, 
based  around  a  tactical-level  comparison  of  forces.  Owing  to  the  Soviet  numerical 
advantage  at  the  tactical  level.  Active  Defense  essentially  came  down  to  preparations 
to  “fight  to  win  a  draw.”  CNA  analyst  Douglas  Skirmer  criticizes  this  concept  as  the 


24.  Paul  Bracken,  "Net  Assessment:  A  Practical  Guide,"  Parameters  36,  no.  1  (Spring  2006):  p. 
100. 

25.  To  gain  useful  perspective  on  the  consideration  of  military  effectiveness  in  the  1980s,  see 
AUan  R.  MUlett,  WilUamson  Murray,  and  Kermeth  H.  Watman,  "The  Effectiveness  of  Mihtary 
Organizations,"  International  Security  11,  no.  1  (Summer  1986).  Here,  the  authors  observe  the 
lack  of  clarity  m  defining  military  effectiveness  and  provide  a  conceptualization  of  how  to 
approach  it  at  the  poUtical,  strategic,  operational,  and  tactical  levels.  However,  “effectiveness” 
throughout  the  work  refers  to  how  well  the  force  measures  up  to  an  ideal  version  of  itself, 
rather  than  whether  it  meets  its  objectives. 

26.  The  JMNA  was  a  net  assessment  prepared  by  the  Chairman  of  the  Joint  Chiefs  of  Staff. 
Closely  tied  to  the  defense  budget,  it  compared  defense  capabiUties  and  programs  of  the  U.S. 
and  its  alUes  to  their  adversaries.  It  was  very  influential  m  the  years  leading  up  to  the  collapse 
of  the  Soviet  Union,  but  lost  relevance  once  the  Warsaw  Pact  dissolved. 

27.  U.S.  Department  of  Defense,  1990  Joint  Military  Net  Assessment  (Washington,  DC:  U.S. 
Department  of  Defense,  1990),  p.  1-2. 
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“result  of  a  doctrinal  vacuum”  that  followed  the  Vietnam  War.^**  U.S.  military  leaders 
had  been  engrossed  in  that  conflict  for  so  long  that  it  had  come  to  define  their 
thinking  at  the  expense  of  other  very  pressing  issues,  leaving  them  with  a  weak 
intellectual  framework  for  facing  a  formidable  foe.  In  this  envirorrment,  where 
objectives  were  defensively  oriented,  operations  assessment  was  considered  largely 
irrelevant.  The  zero  sum  Cold  War  mentality  prevailed:  either  the  U.S.  and  its  allies 
kept  the  Soviets  at  bay,  or  they  didn’t.^'* 

By  the  late  1970s  and  early  1980s,  the  post-Vietnam  malaise  had  subsided  to  a 
degree  and  doctrine  had  evolved  from  Active  Defense  to  Airland  Battle.^”  This  shift 
brought  a  much  more  assertive  orientation  to  U.S.  military  doctrine.  Airland  Battle 
lifted  the  focus  from  the  tactical  to  the  operational  level.  It  emphasized  flexibility 
and  outmaneuvering  the  enemy,  even  in  the  face  of  numeric  disadvantage,  rather 
than  the  direct  quantitative  comparison  of  forces  and  capabilities.^^  Still,  operations 
assessment  continued  to  play  a  minor  role  and  was  a  responsibility  largely  relegated 
to  the  commander.  Army  Field  Manual  100-5  (from  1982)  characterizes  that 
responsibility  as  follows: 

To  insure  [sic]  that  his  concept  is  driving  the  operation  to  completion, 
the  commander  must  follow  up.  He  must  have  an  overall  view  of  the 
battle  based  on  reports  from  subordinates,  information  from 
surveillance  systems,  and  his  own  personal  observations.  He  must 
compare  enemy  and  friendly  situations,  assess  progress,  and  if 
necessary,  issue  fragmentary  orders  (FRAGO)  to  adjust  assigned 
tasks.  This  process  of  follow-up  and  reassessment  is  one  of  the 
commander’s  critical  responsibilities.^^ 

Doctrine  on  the  “command  estimate”  also  provides  useful  insight  into  the  late  Cold 
War  perspective  on  operations  assessment.  Like  most  guidance  on  assessments,  it 
resides  in  planning  publications.  One  student  of  the  planning  process  in  1990 
describes  the  command  estimate  as  a  “logical  and  orderly  examination  of  all  factors 
affecting  the  accomplishment  of  the  mission.”^^  He  goes  on  to  characterize  it  as  a 
“continuing  mental  process  for  the  commander  ...  who  must  observe,  evaluate. 


28.  Douglas  W.  Skinner,  Airland  Battle  Doctrine,  CNA  Professional  Paper  463  (1988),  p.  9. 

29.  Ibid.,  p.  1. 

30.  For  a  detailed  look  at  Airland  Battle,  see  U.S.  Department  of  the  Army,  Operations,  Field 
Manual  100-5  (Washington,  DC:  U.S.  Department  of  the  Army,  1982). 

31.  Skiimer,  Airland  Battle  Doctrine,  p.  1. 

32.  U.S.  Department  of  the  Army,  Operations,  pp.  2-7  -  2-8. 

33.  Timothy  R.  Puckett,  "Rule-Based  Expert  Systems  in  the  Command  Estimate:  An  Operational 
Perspective"  (U.S.  Army  Command  and  General  Staff  College,  1990),  p.  21. 
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revise,  decide,  and  observe  again”  throughout  an  operation.^"*  The  commander’s 
estimate  comprised  the  most  comprehensive  doctrinal  guidance  on  operations 
assessment  at  that  time.  Again,  because  the  U.S.  was  not  engaged  in  any  large-scale 
or  prolonged  armed  conflicts,  demand  for  assessments  was  minimal. 


Development  of  Soviet  Doc  trine 


Also  of  note  during  the  Cold  War  were  the  concurrent  developments  in  Soviet 
doctrine  and  theory.  The  Soviets  had  devoted  significant  resources  to  operations 
research  since  the  early  days  under  Vladimir  Lenin,  who  encouraged  the  use  of 
scientific  methods  in  understanding  warfare.  Subsequent  changes  to  doctrine  in  the 
1920s  and  ‘30s  were  based  on  highly  scientific  systems  theory.^^  That  emphasis 
continued  to  have  influence  on  Soviet  doctrine  throughout  the  Cold  War. 

The  concept  of  correlation  of  forces  and  means  (COFM)  is  one  example  of  this 
effect.’®  Essentially,  COFM  (sometimes  referred  to  simply  as  correlation  of  forces,  or 
COF)  seeks  to  compare  the  relative  power  of  one  military  force  against  another,  using 
rigorous  quantitative  methods.  It  became  especially  prevalent  in  Soviet  military 
dogma  in  the  1970s  because  the  USSR  understood  and  sought  to  exploit  their 
quantitative  tactical  advantage  over  the  United  States.’^  The  influence  of  this  concept 
on  U.S.  doctrine  is  apparent  in  areas  like  net  assessment  and  more  generally  in  the 
focus  on  comparing  forces.  Still,  like  their  U.S.  counterparts,  Soviet  military  thinkers 
devoted  their  energy  to  assessing  friendly  force  capabilities,  rather  than  measuring 
progress  toward  meeting  identified  objectives  during  an  operation.’® 


34.  Ibid.  Views  on  the  command  estimate  varied  in  the  i980s.  Some  proponents  of  net 
assessment  tended  to  view  it  as  too  formulaic  and  short-sighted  (see  Cohen,  "Net  Assessment: 
An  American  Approach,"  pp.  7-8.).  Others  thought  its  consideration  of  qualitative  and 
quantitative  inputs  left  too  much  room  for  bias  and  subjectivity.  As  automation  and  computing 
capabilities  advanced,  so  did  recommendations  to  apply  these  advantages  to  military  decision 
making  processes  to  help  compensate  for  shortfalls  (see  Puckett,  "Rule-Based  Expert  Systems  in 
the  Command  Estimate:  An  Operational  Perspective,"  pp.  21-24.). 

35.  Justui  Kelly  and  David  Kilcullen,  "Chaos  Versus  Predictability:  A  Critique  of  Effects  Based 
Operations,"  Australian  Army  Journal  2,  no.  1  (2006). 

36.  James  K.  Womack,  "Soviet  Correlation  of  Eorces  and  Means:  Quantifying  Modern 
Operations"  (US  Army  Command  and  Staff  College,  1990). 

37.  Mackubin  T.  Owens,  "The  Correlation  of  Forces:  Then  and  Now,"  Ashbrook  (February  2004). 

38.  COFM  found  its  way  Into  U.S.  miUtary  parlance  as  well.  At  the  strategic  level,  it  is  essentially 
analogous  to  the  Western  concept  of  “balance  of  power.”  In  its  apphcation  at  the  tactical  level, 
it  involves  the  effort  to  assess  systematically  one’s  own  combat  power  relative  to  the  adversary, 
which  is  commonly  found  hi  contemporary  U.S.  military  doctrine.  (See  Richard  E.  Porter, 
"Correlation  of  Forces:  Revolutionary  Legacy,"  Air  University  Review  (March-April  1977).) 
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Implic  ations  for  Operations  Assessment 


The  lack  of  a  deliberate  and  detailed  approach  to  operations  assessment  during  the 
Cold  War  era  can  be  attributed  to  two  major  factors.  The  first  was  the  persistent 
potential  for  conflict  with  the  USSR  that  consumed  the  minds  of  Cold  War  leaders. 
U.S.  defense  policy  was  based  almost  entirely  on  how  its  own  forces  and  capabilities 
stacked  up  against  the  adversary.  All  resources  and  faculties  concentrated  on 
deterring  conflict,  and  so  little  thought  was  given  to  how  to  assess  it  once  it  began. 
The  destructive  power  of  nuclear  weapons  and  the  implications  of  their  use  (e.g., 
Mutually  Assured  Destruction)  also  made  the  idea  of  measuring  success  on  the 
battlefield  seem  largely  unimportant.  As  such,  there  was  little  demand  for  a  rigorous 
process  to  structure  assessments. 

The  second  factor  contributing  to  the  relative  obscurity  of  operations  assessment 
was  the  forsaking  of  the  quantitative  measures  of  Vietnam  for  the  more  balanced 
approach  of  relying  on  commanders  to  use  experience,  feedback  from  subordinates, 
and  data  from  the  battlefield  to  build  assessments  around  a  loose  set  of  guidelines. 
With  much  of  the  onus  on  the  commander,  there  was  no  significant  push  for  a  more 
robust  assessment  process.  However,  in  the  years  ahead,  the  collapse  of  the  Soviet 
Union  and  the  rise  of  the  Information  Age  would  bring  a  new  perspective  and 
renewed  emphasis  on  operations  assessment  in  the  U.S.  armed  forces. 


Comparing  Soviet  doctrine  to  that  of  the  West,  an  Army  Command  and  General  Staff  CoUege 
student  in  1990  writes  that  “the  Soviet  COFM  methodology  has  some  distinct  merits  which 
ought  to  attract  the  attention  of  Western  armies,  who  typicaUy  eschew  numerical  methods  in 
mihtary  decisionmaking.”  (Womack,  "Soviet  Correlation  of  Forces  and  Means:  Quantifying 
Modern  Operations,"  p.  96.) 
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Ihe  Infbmnation  Age 


With  the  dramatic  conclusion  of  the  Cold  War,  the  United  States  found  itself  the  only 
remaining  superpower  in  an  era  that  was  filled  with  opportunities.  One  of  the  most 
important  concurrent  developments  was  the  emergence  of  the  so-called  “Information 
Age.”  Computing  capabilities  were  increasing  exponentially  in  very  short  periods  of 
time  and  their  reach  was  extending  into  all  aspects  of  American  life.  Transformation 
on  a  large  scale  was  occurring  across  all  sectors  of  society,  and  the  Pentagon  was 
eager  to  leverage  its  dominant  global  position  by  exploiting  new  technologies  to  stay 
ahead  of  any  competitors  that  might  arise.  While  we  are  ultimately  concerned  with 
how  Information  Age  developments  affected  operations  assessment,  first  we  will 
briefly  examine  the  major  theoretical  undercurrents  of  the  time  to  provide  context.^® 


Conceptual  Tiends 

Revolution  in  Military  Affairs  (RM A) 

The  Revolution  in  Military  Affairs  broadly  refers  to  the  changes  that  technological 
advancements  have  brought  to  modern  warfare  across  a  wide  swath  of  systems, 
areas,  and  issues.  U.S.  Navy  Admiral  William  (Bill)  Owens,  who  has  written  extensively 
on  the  RMA  concept,  concedes  that  it  is  difficult  to  pinpoint  exactly  when  the 
revolution  began.  Rumblings  of  a  major  overhaul  in  the  way  the  United  States  fights 
its  wars  were  apparent  as  early  as  the  1960s,  though  the  RMA  is  typically  associated 
with  the  post-Cold  War  era."*®  Owens  groups  developments  like  AdrLand  Battle  and 
advances  in  communications  technology  under  the  general  heading  of  the  early  days 
of  RMA.  Following  Operation  Desert  Storm,  the  potential  of  this  revolution  became 
increasingly  apparent  and  “advantages  of  precision,  reach,  battlespace  awareness, 
space-based  observation,  and  advanced  communications  became  prominent  in  the 


39.  Note  that  while  they  are  addressed  individually,  there  is  significant  overlap  among  these 
concepts  and  trends,  both  in  content  and  in  timing. 

40.  WiUiam  A.  Owens,  "The  Once  and  Future  Revolution  in  Military  Affairs,"  Joint  Forces 
Quarterly,  no.  31  (2002). 
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Armed  Forces.”"*'  With  the  RMA,  capabilities  like  speed,  agility,  and  precision  replaced 
a  long-held  emphasis  on  toe-to-toe  engagements. 

Also  important  to  the  RMA  was  the  return  of  systems  analysis  to  prominence  after 
its  influence  diminished  in  the  post-Vietnam  years.  Most  often  referred  to  as  a 
“system  of  systems”  approach  in  the  context  of  RMA,  this  take  on  decades-old 
methods  emphasizes  the  interconnectedness  of  modern  military  operations  and  the 
need  to  exploit  the  potential  of  a  number  of  advanced  systems  simultaneously  (e.g., 
precision  guided  weapons,  advanced  intelligence,  surveillance,  and  recormaissance 
(ISR),  and  command,  control,  communications,  computers  (C4)).‘‘*'  What  results  is  a 
whole  that  is  greater  than  the  sum  of  its  parts,  able  to  carry  out  operations  and  meet 
objectives  with  greater  efficacy  and  efficiency  than  the  systems  could  achieve 
separately. 

Network-Centric  Warfare  (NCW) 

Closely  related  to  the  RMA  is  the  concept  of  network-centric  warfare  (NCW),  which 
emerged  around  the  same  time  in  the  mid-1990s.  Admiral  Arthur  Cebrowski,  a 
prominent  advocate  of  NCW,  sums  it  up  by  stating,  “NCW  is  not  narrowly  about 
technology,  but  broadly  about  an  emerging  military  response  to  the  information 
age.”"*’  As  such,  it  encompasses  more  than  advanced  computing  and  greater 
networkability  (though  these  are  central  to  NCW).  Cebrowski  distills  the 
characterization  of  NCW  to  three  main  points:  networks  take  precedence  over 
platforms;  actors  are  viewed  in  the  context  of  the  system  in  which  they  operate;  and 
adaptabihty  becomes  paramount."*^ 

The  origins  of  NCW  exist  in  part  in  the  defense  industry  itself,  where  concepts  like 
the  Internet  first  developed.  However,  NCW  owes  much  of  its  genesis  and  subsequent 
development  to  the  economic  sector.  By  the  1990s,  the  concept  of  “network-centric 
computing”  had  revolutionized  the  way  businesses  operated.  The  rntercormectedness 
of  the  various  nodes  of  a  business’  “ecosystem”  enabled  real-time  information 
sharing  and  analysis  that  dramatically  increased  the  pace  of  operations.  Moving 
faster  gave  pioneering  businesses  a  distinct  advantage  over  their  competitors. 
Defense  industry  leaders  sought  to  replicate  those  benefits  and  exploit  the 
warfighting  advantages  of  network  centricity.  Soon  the  language  and  methods  of  the 
Information  Age  were  appearing  across  the  defense  industry.  Phrases  like  “network- 


41.  Ibid. 

42.  'The  Emerging  System  of  Systems,"  Military  Review  75,  no.  3  (1995). 

43.  Arthur  K.  Cebrowski,  "Network-Centric  Warfare,"  Military  Technology  27,  no.  5  (2003). 

44.  "Network-Centric  Warfare:  Its  Origin  and  Future,"  U.S.  Naval  Proceedings  124  (January 
1998). 
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centric  operations”  and  “competitive  advantage”  over  an  adversary,  typically  found  in 
business  parlance,  became  common  in  defense  circlesd^  As  for  operations 
assessment,  the  economic  sector  had  demonstrated  the  potential  for  advanced  data 
gathering  and  processing  capabilities  to  provide  timely  and  accurate  measures  of 
success.  Expectations  that  a  similar  approach  could  be  applied  in  the  context  of 
military  operations  began  to  emerge  and  gain  influence. 

Effects- Based  Operations (EBO) 

These  pivotal  shifts  in  thinking  on  warfare  produced  yet  another  important  concept 
that  would  have  significant  implications  for  operations  assessment.  The  idea  of 
effects-based  operations  (EBO)  surfaced  at  the  same  time  as  trends  like  RMA  and 
NCW,  but  was  focused  on  the  application  of  the  new  capabilities  brought  on  by  tbe 
Information  Age,  ratber  than  tbe  capabilities  themselves.  Instead  of  measuring 
success  through  targeting  and  inflicting  damage  on  the  enemy  for  its  own  sake,  EBO 
attempts  to  achieve  a  desired  effect  by  using  the  right  stimulus  to  create  the  desired 
response."*®  EBO  approaches  developed  in  the  1990s  were  largely  focused  on  warfare 
against  states  and  emphasized  the  conduct  of  “Systems  of  Systems  Analysis”  (SOSA) 
or  “Operational  Net  Assessments”  (ONA)  as  the  initial  steps  in  planning  to  achieve 
effects.  These  activities  consisted  of  identifying  and  analyzing  the  nodes  and 
cormections  within  and  amongst  enemy  systems  in  an  effort  to  identify  those  that,  if 
destroyed,  would  create  the  maximum  desired  effect  on  the  enemy  (perhaps  while 
also  minimizing  any  undesired  effects). 

Assessing  the  success  of  effects-based  operations  required  the  development  of  new 
ways  of  measuring  progress  toward  achieving  a  commander’s  objectives,  which  gave 
rise  to  the  development  of  a  new  assessment  framework  beyond  traditional  methods 
like  the  command  estimate  or  battle  damage  assessment  (BDA).  Labeled  effects-based 
assessment  (EBA),  this  framework  would  comprise  most  of  the  guidance  on 
assessments  as  the  U.S.  headed  into  the  wars  in  Iraq  and  Afghanistan.  Central  to  EBA 
is  the  idea  of  using  measures  of  effectiveness  (MOE)  and  measures  of  performance 
(MOP)  to  evaluate  whether  the  commander’s  objectives  are  being  met  and  associated 
tasks  are  being  completed,  respectively.  In  this  framework,  MOPs  and  MOEs  would 
naturally  fall  out  of  the  SOSA  or  ONA  activities,  since  presumably  once  the  enemy’s 
systems  were  understood,  the  targeting  and  destruction  of  key  nodes  and 
cormections  would  produce  effects  that  could  be  predicted  and  therefore  measured. 


45.  For  example,  see  U.S.  Department  of  Defense,  The  Implementation  of  Network-Centric 
Warfare  (Washington,  DC:  U.S.  Department  of  Defense,  2005). 

46.  Edward  A.  Smith,  Effects  Based  Operations:  Applying  Network  Centric  Warfare  in  Peace, 
Crisis,  and  Mhr  (Command  and  Control  Research  Program,  2002),  p.  xiv. 
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Like  NCW,  EBO  mirrors  concepts  embraced  in  the  business  world.  The  idea  of 
measuring  outcomes,  rather  than  inputs  and  processes,  took  the  form  of  what  is 
known  as  results-based  management  (RBM),  which  came  to  prominence  in  American 
business  practices  in  the  1980s.'‘^  Under  RBM,  measuring  success  in  management  was 
based  on  the  outcomes,  rather  than  the  outputs,  produced.  Peter  Druker,  a  prolific 
writer  on  management  and  an  RBM  advocate,  states  simply,  “A  business  management 
has  failed  if  it  fails  to  produce  economic  results.”'*®  In  1991,  strategists  in  Desert 
Storm  built  a  similar  approach  to  measuring  success  in  a  military  context.  Its 
ultimate  focus  was  on  the  results  (or  effects)  of,  for  example,  destroying  a  target 
rather  than  the  destruction  itself.  The  EBO  concept  continued  to  gain  influence  and 
advocates  after  Desert  Storm  and  was  eventually  incorporated  into  official  U.S. 
military  doctrine. 

These  developments  represented  the  most  robust  attempt  at  developing  a 
quantitative  operations  assessment  process  since  Vietnam.  General  James  Maths,  in  a 
famous  memo  to  his  subordinates  at  the  United  States’  Joint  forces  Command 
(JECOM),  referred  to  this  shift  as  a  “renaissance  in  combat  assessment  beyond  simple 
battle  damage  assessment.”***  However,  as  we  will  see  in  the  next  section,  the  EBO 
concept  was  unceremoniously  killed  in  the  same  memo,  leaving  a  vacuum  of 
operations  assessment  doctrine  in  the  midst  of  two  major  wars. 


Ihe  NATO  Experience 


NATO’s  involvement  in  the  Balkans  in  the  1990s  offers  a  helpful  case  study  for 
operations  assessment  after  the  Cold  War.  Because  NATO  was  preoccupied  with 
countering  the  Soviet  Union  and  the  Warsaw  Pact  for  decades,  the  alliance  did  not 
participate  in  actual  military  operations  until  the  Cold  War  had  ended.  In  1992, 
NATO  became  engaged  in  the  former  Yugoslavia  in  what  would  become  its  first 
major  military  operation.  The  NATO  experience  in  Bosnia  and  Herzegovina  (BiH) 
began  with  an  air  campaign,  but  the  bulk  of  the  assessments  work  occurred  in  the 
years  of  stabilization  that  followed  the  Dayton  Peace  Accords  in  December  1995.  The 
Implementation  force  (IfOR)  and  subsequent  Stabilization  force  (SfOR)  employed  an 
“Operational  Analysis”  (OA)®"  office  to  conduct  assessments  that  measured  progress 

47.  John  C.  Morris  and  Andrew  P.  WiLLiams,  "The  Development  of  Theory-Driven  Evaluation  in 
the  Mihtary,"  American  Journal  of  Evaluation  30,  no.  1  (March  2009). 

48.  Peter  F.  Druker,  Management:  Tasks,  Responsibilities,  Practices  (New  York:  Truman  Tatley 
Books,  1986),  p.  33. 

49.  James  N.  Mattis,  "U.S.  Joint  Forces  Command  Commander's  Guidance  for  Effects-Based 
Operations,"  Parameters  38,  no.  3  (Autumn  2008). 

50.  in  this  context,  operations  analysis  is  analogous  with  operations  research. 
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on  the  ground.  Their  experience  afforded  several  important  lessons  in  the  field  of 
operations  assessment.  We  will  highlight  some  of  the  most  notable  findings  from 
those  operations.” 

First,  it  is  important  to  note  that  the  NATO  assessment  process  was  not  established 
prior  to  its  involvement  in  the  Balkans.  Owing  to  the  nature  of  the  operation — 
stabilization  rather  than  offensive  operations — “few  of  the  standard  military  OA 
tools  could  be  profitably  applied  to  the  circumstances  of  BiH.””  Analysts  had  to 
create  a  process  through  trial  and  error  in  real  time.  Consequently,  the  organization 
and  methods  used  to  do  assessments  were  (by  necessity)  very  flexible  and  were 
adjusted  fairly  often  to  compensate  or  account  for  shortfalls.” 

Analysts  took  a  very  scientific  approach  to  this  development  process,  relying  on 
quantitative  data  with  qualitative  interpretation.  Collectors  would  gather  data  and 
analysts  would  use  their  expertise  to  interpret  them  and  make  necessary  changes, 
assign  scores,  and  present  their  findings.  A  significant  portion  of  the  data  came  from 
surveys  that  NATO  forces  conducted  and  from  interaction  with  civilian  aid 
organizations  and  other  similar  bodies.”  To  aggregate  and  report  the  data,  analysts 
developed  a  scoring  system,  based  on  “carefully  worded  scoring  criteria,”  which 
resembled  the  “stoplight  charts”  that  would  become  common  in  assessments  in  the 
Iraq  and  Afghanistan  theaters.”  They  also  used  the  language  “measures  of 
effectiveness”  and  “measures  of  success”  in  their  assessments,  reflecting  the 
influence  of  EBO.  Because  the  approach  leaned  heavily  on  mathematical  methods. 


51.  For  a  detailed  treatment  of  this  topic,  see  Nicholas  J.  Lambert,  "Measuring  the  Success  of 
the  Nato  Operation  in  Bosnia  &  Herzegovina  1995-2000,"  in  17th  European  Conference  on 
Operations  Research  (Budapest,  Hungary,  2001);  Stephan  B.  Flemming  and  WiUiam  J.  Owen, 
Measurement  of  Post-Conflict  Resolution  in  Bosnia-Herzegovina,  ed.  Alexander  E.  R.  Woodcock 
and  David  F.  Davis,  The  CornwalUs  Group  VI:  Analysis  for  Assessment,  Evaluation  and  Crisis 
Management  (Clementsport,  N.S.:  Canadian  Peacekeeping  Press,  2002);  and  Suzarme  Griffin, 
"Operational  Analysis  at  the  Erontllne  -  a  Generic  Approach  to  Measuring  Progress?,"  in 
1 71SMOR,  Session  J  (2000). 

52.  Flemming  and  Owen,  Measurement  of  Post-Conflict  Resolution  in  Bosnia-Herzegovina,  p.  347. 

53.  The  first  iteration  of  assessments  measuring  progress  in  NATO  operations  in  BiH  Involved 
the  use  of  “Normality  Indicators,”  which  analysts  built  to  measure  how  close  to  normal 
different  areas  were.  (“Normal”  was  based  on  a  “control”  town  that  had  been  left  largely 
untouched  by  the  recent  violence.)  Next,  analysts  used  the  “Measures  of  Effectiveness”  process, 
followed  by  the  “Six-Month  Review.”  (Griffin,  "Operational  Analysis  at  the  Frontline  -  a  Generic 
Approach  to  Measuring  Progress?,"  p.  3.)  Each  of  these  approaches  was  based  on  several 
factors,  including  the  nature  of  NATO  operations  at  the  time,  the  security  situation  on  the 
ground,  and  the  types  of  data  that  were  of  most  Interest. 

54.  Lambert,  "Measuring  the  Success  of  the  Nato  Operation  m  Bosnia  &  Herzegovina  1995- 
2000." 

55.  Ibid.,  p.  465. 
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one  of  the  recommendations  coming  out  of  NATO’s  experience  in  the  Balkans  was  for 
better  automation  and  computing  capabilities  to  enable  a  more  systematic  way  to 
deal  with  the  large  amounts  of  data.^® 

NATO’s  dedication  to  developing  a  workable  operations  assessment  process  has 
continued  since  the  conclusion  of  operations  in  the  Balkans,  first  (and  somewhat 
simultaneously)  in  Kosovo, and  then  in  the  NATO  commitment  to  Afghanistan  from 
2003  to  the  present.  EBO  has  been  particularly  influential  in  NATO’s  development  of 
an  operations  assessment  process,  and  that  influence  continued  even  after  U.S. 
defense  leaders  largely  rejected  the  concept  and  its  application  to  assessments.  We 
will  discuss  these  developments  in  the  next  section. 


Implic  ations  for  Opeiations  Assessment 


The  major  undercurrents  in  the  transformation  that  occurred  in  the  defense  industry 
after  the  Cold  War  had  a  marked  impact  on  the  way  the  armed  forces  approached 
operations  assessment.  Simply  put,  military  leaders  posited  that  the  Information  Age 
brought  with  it  the  ability  to  do  what  previous  generations  had  found  untenable: 
leverage  vast  information  gathering  and  computing  capabilities  to  make  precise, 
nuanced,  and  quantified  measurements  of  developments  on  the  battlefield.  Rather 
than  relying  on  “intuition  and  ad  hoc  metrics,”  commanders  and  their  subordinates 
could  now,  in  theory,  build  more  informed  and  objective  assessments.^** 

Military  doctrine  predictably  reflected  these  fundamental  changes  in  thought  and 
theory.  In  addition  to  the  impact  of  the  Information  Age,  the  rapid  disintegration  of 
the  Cold  War  bipolar  paradigm  that  had  shaped  defense  policy  for  decades  signaled  a 
huge  shift  in  doctrine,  which  had  to  evolve  to  keep  pace  with  new  thinking  on  a  21st 
century  conception  of  warfare.**®  The  Joint  Operations  Planning  and  Execution  System 
(JOPES)  provides  a  clear  example  of  the  changes  occurring  in  doctrine  and  specifically 
in  operations  assessment.****  The  1996  version  of  JOPES  refers  to  assessments  in  very 


56.  Ibid.,  p.  478. 

57.  For  more  details  on  NATO  assessments  in  Kosovo,  see  M.R.  Neighbour  et  al.,  "Providing 
Operational  Analysis  to  a  Peace  Support  Operation:  The  Kosovo  Experience,"  Journal  of  the 
Operational  Research  Society  53,  no.  5  (2002). 

58.  Smith,  Effects  Based  Operations:  Applying  Network  Centric  Warfare  in  Peace,  Crisis,  and 
War,  p.  356. 

59.  While  doctrinal  changes  were  manifold,  the  focus  here  is  limited  to  changes  in  operations 
assessment. 

60.  As  mentioned  previously,  doctrinal  discussion  on  operations  assessment  is  typically  nested 
in  the  planning  process. 
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general  terms  with  little  specific  guidance.  However,  with  the  1999  version  came 
much  more  detail,  including  language  like  “measures  of  merit”  to  assist  plarmers  in 
assessing  how  well  forces  were  doing  at  meeting  the  commander’s  objectives.  This 
more  quantitative,  systems-based  approach  reflects  the  growing  influence  of  EBO  and 
similar  concepts  on  doctrine  and  practices  in  the  late  1990s— concepts  that  promised 
clarity  and  predictability  of  outcomes  in  warfare.  Their  impact  on  operations 
assessment  would  become  even  more  profound  when  the  U.S.  became  engaged  in  two 
simultaneous  counterinsurgencies. 
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Ihe  Post-Q/ 11  Eia 


The  attacks  by  A1  Qaeda  against  the  United  States  on  11  September  2001  and  the 
Global  War  on  Terror  that  ensued  ushered  in  a  new  era  of  United  States  and  coalition 
warfighting  that  is  still  ongoing.  In  addition  to  pursuing  A1  Qaeda  and  its  affiliates 
via  small  footprint  engagements  across  the  globe,  the  United  States  conducted 
regular  and  irregular  major  combat  operations  against  the  governments  of  Iraq  and 
Afghanistan,  respectively,  leading  directly  to  the  overthrow  of  the  ruling 
establishments  in  those  countries  and  prolonged  insurgencies  that  continue  to  this 
day.  Developing  a  workable  operations  assessment  framework  in  these  conflicts  has 
proven  difficult,  just  as  it  was  for  Defense  leaders  in  Vietnam. 

Coming  out  of  the  Information  Age,  operations  assessment  was  continuing  its 
evolution  back  to  an  emphasis  on  quantitative  and  systems  analysis  tools  and 
methods,  as  evidenced  by  the  prevalence  of  EBQ  and  its  associated  analytic 
techniques,  SQSA  and  QNA.  This  shift  continued  in  the  early  years  of  the  wars  in  Iraq 
and  Afghanistan.  The  preference  for  scientific  and  mathematical  approaches  among 
many  assessment  groups  (which  were  often  staffed  by  QRSA  persormel)  and  an 
appetite  for  hard  numbers  from  U.S.  military  and  policy  leaders  created  a  situation 
where  quantitative  metrics  dominated  the  formation  of  centralized  operations 
assessments.  However,  as  the  wars  progressed,  the  doctrinal  death  of  EBQ  and 
dissatisfaction  with  the  results  of  quantitative  assessments  in  Iraq  and  Afghanistan 
caused  the  pendulum  of  operations  assessment  to  swing  back  towards  a  greater 
emphasis  on  qualitative  inputs. 


Ihe  Warin  Imq 


Qperation  Iraqi  freedom  began  on  20  March  2003,  with  the  United  States  leading 
major  combat  operations  against  the  government  of  Saddam  Hussein.  The 
conventional  nature  and  speed  of  those  operations  meant  that  assessments  during 
the  invasion  were  focused  on  the  location  and  conditions  of  various  fighting  units  as 
well  as  collating  and  analyzing  BDA  reports  to  show  progress  relative  to  the 
operational  plan.  In  the  wake  of  the  collapse  of  Saddam  Hussein’s  government,  the 
coalition,  led  by  the  United  States,  was  slow  to  transition  from  major  combat  to 
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stability  operations,  in  part  because  policymakers  had  forbidden  military  plarmers 
from  engaging  in  detailed  planning  for  the  post-conflict  phase.®^  In  the  years  that 
followed,  a  virulent  insurgency  developed  and  military  forces  from  the  United  States 
and  its  coalition  partners  engaged  in  (or  supported)  counterinsurgency  operations  in 
Iraq  until  the  end  of  2011.®^ 

From  2004  to  2006,  the  gradual  shift  in  United  States  military  operations  to  a  focus 
on  counterinsurgency,  combined  with  the  continuing  prevalence  of  EBO  and  its 
quantitative  approach  to  assessment,  resulted  in  a  sustained  emphasis  on  the 
development  and  use  of  quantitative  metrics  for  assessing  progress  in  the  war.®®  As  it 
became  clear  that  the  growing  insurgency  would  require  significant  effort  to  quell, 
operations  assessment  became  increasingly  significant.  Methods  used  to  build 
assessments  varied  widely,  but  many  included  the  use  of  indicators®"*  plotted  over 
time  to  identify  trends.  From  these  data,  assessors  could  extrapolate  MOEs  and 
MOPs,  in  keeping  with  EBA  doctrine.  Frequently  used  indicators  included  the  number 
and  quality  (i.e.,  how  well  trained  and  equipped)  of  Iraqi  security  forces,  which 
proved  to  be  extremely  difficult  to  measure.®®  Other  countrywide  assessment 
processes  were  developed  to  measure  progress  in  areas  such  as  security,  governance, 
and  the  provision  of  basic  essential  services  (e.g.,  water,  electricity,  sanitation).®®  In 
some  cases,  indicators  were  assigned  weights  (expressed  numerically),  which  were 
then  aggregated  and  averaged  to  characterize  progress. 

Other  methods  that  became  more  common  around  2007  included  color-coded  charts 
(often  called  “stoplight  charts”  for  the  frequently  used  green/amber/red 


61.  Michael  Gordon  and  Bernard  Trainor,  Cobra  11:  The  inside  Story  of  the  Invasion  and 
Occupation  of  Iraq  (New  York:  Vintage,  2006). 

62.  As  this  paper  was  being  written,  the  U.S.  and  several  other  nations  were  sending  military 
advisors  to,  and  conducting  airstrikes  in,  Iraq  to  counter  a  terrorist  organization  called  the 
“Islamic  State,”  which  grew  out  of  the  primary  Surmi  insurgent  group  in  the  country— al  Qaeda 
in  Iraq. 

63.  Unfortunately,  the  details  of  operations  assessment  processes  and  products  from  this  time 
period  in  the  Iraq  war  are  not  well  documented.  Much  of  what  follows  regarding  the  operations 
assessment  process  in  Iraq  is  based  on  the  authors’  observations  and  experience  in  al  Anbar 
province  from  2007-2009,  as  weU  as  on  a  number  of  informal  conversations  with  military 
members  who  served  in  Iraq  during  this  tune. 

64.  Indicators  were  usually  derived  from  MOEs  and  MOPs,  and  together  they  were  considered 
the  “metrics”  used  hi  building  assessments.  We  will  discuss  the  issues  associated  with  choosing 
metrics  below. 

65.  Frederick  W.  Kagan,  "Measuring  Success,"  Armed  Forces  Journal  (January  2006). 

66.  Center  for  Strategic  and  International  Studies,  "Progress  or  Peril?  Measuring  Iraq's 
Reconstruction,"  (September  2004). 
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designations).®^  Stoplight  charts  took  different  forms,  but  they  typically  used  a  single 
color  to  represent  the  state  of  an  indicator  or  measure  (e.g.,  areas  with  high  levels  of 
enemy  activity  would  be  designated  as  “red”).  While  this  approach  allowed 
assessment  teams  to  account  for  more  qualitative  factors,  it  was  difficult  to 
communicate  sufficient  detail  in  a  single  color.  Much  of  the  data  and  judgment  used 
to  generate  that  color  designation  remained  opaque  to  the  audience.  Overall, 
however,  operations  assessment  continued  to  emphasize  the  importance  of 
quantifying  indicators  and  plotting  trends  over  time. 

To  enable  the  quantitative  approach  to  assessments,  a  central  repository  was  created 
for  the  data  that  would  be  used  to  calculate  quantitative  MOEs  and  MOPs.  In  late 
2006  and  early  2007,  there  was  a  concerted  effort  on  the  part  of  the  strategic  and 
operational  commands  in  Iraq  (Multi-National  Force-Iraq,  and  Multi-National  Corps- 
Iraq,  respectively)  to  consolidate  ah  of  the  military’s  operational  reports  into  a  single, 
classified  database  (called  the  Combined  Information  Data  Network  Exchange,  or 
CIDNE).®*  Operations  analysts  at  MNE-I  and  MNC-I  led  this  effort,  and  CIDNE 
eventually  became  the  primary  database  for  recording  events  on  the  ground  (along 
with  other  data).  Even  with  this  tool,  however,  making  sense  of  the  vast  amount  of 
data  stored  in  it  was  a  challenge  that  persisted  for  the  duration  of  the  war. 

While  much  of  the  assessment  effort  in  Iraq  was  based  in  quantitative  methods  using 
hard  data  from  sources  like  CIDNE,  there  were  exceptions.  The  assessment  process 
conducted  by  Multi-National  Forces-West  (MNF-W)  in  2007,  for  example,  rehed  on 
more  qualitative  methods  involving  the  use  of  standards.  These  standards  were 
defined  for  a  number  of  “levels”  in  each  of  several  areas  deemed  important  for 
counterinsurgency  efforts  (e.g.,  security,  governance,  essential  services,  and  rule  of 
law),  and  were  applied  to  a  number  of  key  cities  and  towns  in  Anbar  Province.  A 
collaborative,  cross-functional  working  group  met  on  a  weekly  basis  to  produce 


67.  Connable,  "Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency,"  p.  49. 

68.  The  CIDNE  database  was  a  contracted  effort  to  comphe  what  had  previously  been  a 
hodgepodge  of  spreadsheets,  databases,  and  loose  files  into  a  single  repository  of  mihtary 
operations  reports  (so-called  Significant  Activity  (SIGACT)  reports).  Most  operational 
commands,  and  many  operational  units,  in  Iraq  eventuaUy  transitioned  to  reporting  their 
activities  directly  into  CIDNE— though  the  Marines  in  al  Anbar  province  never  did  (they  retained 
their  own  database  which  was  manually  cross-loaded  into  CIDNE  at  MNC-I).  As  a  result,  the 
CIDNE  database  became  rife  with  duplicate  entries,  as  weU  as  incorrect  or  missing  information. 
MNC-I  addressed  this  by  creating  a  data  coUation  cell  to  perform  quality  control  on  the  entries 
in  CIDNE.  Once  a  week,  this  cell  published  an  off-line  repository  of  “scrubbed  and  vetted” 
SIGACTs,  caUed  the  SIGACTS  III  database.  This  became  the  “database  of  record”  for  deployed 
analysts.  Unfortunately,  in  the  wake  of  the  U.S.  withdrawal  from  Iraq,  this  database  appears  to 
have  been  lost,  though  the  raw,  “dirty”  data  still  reside  in  the  CIDNE  database  at  U.S.  Central 
Command.  The  CIDNE  database  was  later  transported  to  Afghanistan  and  served  a  similar 
purpose  there,  though  with  a  much  wider  aperture  of  data  inputs:  it  included  polhng  results, 
atmospherics  and  inteUigence  reports,  and  the  hke. 
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ratings  against  these  standards  via  discussion  and  debate,  and  then  communicated 
the  ratings  directly  to  the  Commanding  General  of  MNF-W.®  EBA  tended  to  devalue 
such  qualitative  inputs  as  necessarily  subjective  and  therefore  less  reliable.  However, 
as  the  limitations  of  effects-based  thinking  became  more  apparent,  there  was  an 
effort  at  some  commands  to  shift  to  a  more  balanced  approach  like  that  used  by 
MNF-W. 

At  the  strategic  level,  though,  the  Iraq  assessments  process  encountered  the  same 
difficulties  and  pitfalls  of  operations  assessment  in  Vietnam.  The  hopes  that  EBO  and 
vastly  improved  technological  capabilities  would  bring  the  same  kind  of  precision  to 
assessments  as  it  had  to  weapons  and  communications  systems  quickly  faded.  Early 
in  the  war,  the  United  States’  political  objectives  and  overall  strategy  were  ill-defined, 
making  it  difficult  to  know  what  to  assess  against.  While  analysts  had  endless 
amounts  of  data  at  their  fingertips,  they  had  no  real  framework  to  help  determine 
how  to  use  it.™  Consequently,  multiple  assessments  emerged  simultaneously  that 
were  neither  coordinated  nor  consistent  with  one  another.^^  As  one  author  wrote, 
“The  media  are  counting  U.S.  casualties.  The  military  counts  Iraqi  soldiers.  Both  are 
measures  of  convenience,  reflecting  the  ease  with  which  data  can  be  collected  and 
presented  rather  than  its  inherent  importance.  Neither  tells  us  whether  we  are 
wirming  or  losing  in  Iraq.”™  By  2006,  this  criticism  of  the  choice  of  metrics  in  Iraq 
assessments  had  become  a  common  theme,  and  would  remain  so  for  the  rest  of  the 
war. 

Even  after  the  United  States  clarified  its  aims  and  strategy  as  part  of  the  so-called 
“surge”  effort,  assessment  processes  continued  to  falter.  They  generally  remained 
very  centralized,  even  though  the  areas  assessed  were  extremely  disparate  and  there 
were  few  mechanisms  to  capture  differences  among  them.  Centralized  data  omitted 
the  nuance  and  context  necessary  to  properly  interpret  conditions  on  the  ground, 
which  is  especially  problematic  in  counterinsurgency.™  In  fact,  metrics  used  in  Iraq 
assessments  were  often  divorced  from  any  meaningful  theory  of  counterinsurgency. 


69.  In  the  interest  of  fuU  disclosure,  one  of  the  authors  (Schroden)  supported  this  assessment 
process  (though  did  not  design  it).  Schroden  was  impressed  by  its  general  simphcity  and 
effectiveness  and  later  transported  it  to  Afghanistan  during  the  redesign  of  ISAF’s  assessment 
process  in  late  2011/early  2012. 

70.  James  Clancy  and  Chuck  Crossett,  "Measuring  Effectiveness  in  Irregular  Warfare," 
Parameters  37,  no.  2  (Summer  2007). 

71.  Craig  Cohen,  "Measuring  Progress  hi  Stabilization  and  Reconstmction,"  hi  Stabilization  and 
Reconstruction  Series  No.  1  (USIP,  March  2006),  p.  4. 

72.  Kagan,  "Measuring  Success." 

73.  Anthony  Cordesman  argues  in  a  2007  article  that  “no  set  of  metrics  is  more  useless  hi 
counterinsurgency  ...  than  national  totals  and  national  averages.”  (See  Anthony  Cordesman, 
"The  Uncertain  "Metrics"  of  Afghanistan  (and  Iraq),"  (May  2007).) 
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and  more  often  than  not  were  calculated  and  analyzed  simply  because  the  capability 
existed  to  do  so.  Compounding  the  problem,  data  were  often  inaccurate  or 
incomplete,  though  aggregation  often  obscured  these  issues  and  gave  the  consumer  a 
greater  sense  of  confidence  in  their  accuracy  than  was  often  warranted.  The  appeal 
of  concrete  numbers  that  helped  drive  the  assessments  process  in  Vietnam 
reappeared  as  leadership  sought  a  clear  picture  of  progress  in  Iraq.  Once  again, 
significant  demand  from  top  policy  leaders  prompted  a  heavy  emphasis  on 
operations  assessments  and  created  pressure  to  produce  accurate  reports. 

Ihe  Death  of  Effects- Based  Opeiations 


As  discussed  earlier,  EBO  and  its  associated  quantitative  assessment  processes  were 
developed  in  the  late  1990’s,  and  the  impact  of  these  concepts  was  clearly  evident  in 
the  Iraq  war.  However,  in  2008,  the  Commanding  General  of  the  United  States  Joint 
Forces  Command  (USJFCOM) — the  organization  responsible  for  the  inclusion  of  EBO 
in  joint  doctrine— issued  a  memo  to  his  subordinates  in  which  he  blasted  the 
fundamental  precepts  on  which  EBO  was  founded,  saying  “it  is  my  view  that  EBO  has 
been  misapplied  and  overextended  to  the  point  that  it  actually  hinders  rather  than 
helps  joint  operations.”  He  listed  his  criticisms  with  EBO  in  order: 

First,  operations  in  the  future  will  require  a  balance  of  regular  and 
irregular  competencies.  Second,  the  enemy  is  smart,  and  adaptive. 

Third,  all  operating  environments  are  dynamic  with  an  infinite 
number  of  variables;  therefore,  it  is  not  scientifically  possible  to 
accurately  predict  the  outcome  of  an  action.  To  suggest  otherwise 
runs  contrary  to  historical  experience  and  the  nature  of  war.  Fourth, 
we  are  in  error  when  we  think  that  what  works  (or  does  not  work)  in 
one  theater  is  universally  applicable  to  all  theaters.  Finally,  to  quote 
Sherman,  “Every  attempt  to  make  war  easy  and  safe  will  result  in 
humiliation  and  disaster.”  History  is  replete  with  such  examples  and 
further  denies  us  any  confidence  that  the  acute  predictability 
promised  by  EBO’s  long  assessment  cycle  can  strengthen  our 
doctrine.^'* 

Near  the  end  of  his  memo.  General  Mattis  provided  the  following  direction  to  the 
United  States  Joint  Force:  “Effective  immediately,  USJFCOM  will  no  longer  use, 
sponsor,  or  export  the  terms  and  concepts  related  to  EBO,  ONA,  and  SOSA  in  our 


74.  Mattis,  "U.S.  Joint  Forces  Command  Commander's  Guidance  for  Effects-Based  Operations." 
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training,  doctrine  development,  and  support  of  JPME  [Joint  Professional  Military 
Education], 

The  impact  of  this  memo  was  instant  and  far-reaching— even  beyond  General  Mattis’ 
intent.  While  he  expressly  stated  his  belief  that  effects-based  targeting  was  a  useful 
concept,  the  weight  and  forcefulness  of  his  memo  effectively  killed  an  approach  to 
planning,  operations,  and  assessment  that  had  been  in  development  and  use  for  over 
a  decade.  However,  the  death  of  EBO  in  name  did  not  effectively  lead  to  the 
elimination  of  many  of  its  precepts  for  operations  assessment.  Rather,  the 
suddenness  of  EBO’s  demise  meant  that  there  were  few  viable  alternative  concepts 
for  operations  assessment,  creating  a  vacuum  that  proved  detrimental  to  assessment 
practices  in  Afghanistan. 

The  impact  of  the  rejection  of  EBO  was  not  as  great  in  Iraq  for  several  reasons.  Eirst, 
OIE  began  as  a  conventional  major  combat  operation,  and  so  the  period  of  time  when 
operations  assessment  processes  focused  on  counterinsurgency  was  comparatively 
short  (approximately  2006-2009^®)  and,  for  the  most  part,  occurred  before  the  end  of 
EBO.  Second,  once  violence  levels  in  Iraq  fell  dramatically  in  2008,  the  demand  for 
operations  assessment  diminished  in  the  United  States. By  2009,  policymakers  were 
preoccupied  with  domestic  concerns  and  no  longer  clamoring  to  know  how  Iraq  was 
progressing.  As  we  will  discuss  in  the  next  section,  however,  the  death  of  EBO  had  a 
much  greater  impact  on  the  war  in  Afghanistan. 


Ihe  War  in  Afghanistan 


On  7  October  2001,  the  United  States  initiated  military  operations  in  Afghanistan  to 
pursue  the  members  of  al  Qaeda  responsible  for  the  attacks  of  If  September  2001 
and  to  remove  the  Taliban  goverrrment  that  had  provided  them  safe  haven.  By 
December  2001,  the  Taliban  and  al  Qaeda  had  largely  fled  Afghanistan.  The  newly 
created  International  Security  Assistance  Eorce  (ISAE)  took  responsibility  for  the 
security  mission  in  Afghanistan  under  the  leadership  of  NATO. 

The  size,  structure,  and  missions  of  ISAE  evolved  and  increased  considerably  over 
the  course  of  2002-2008,  during  which  time  operations  assessment  activities  were 


75.  Ibid. 

76.  By  the  end  of  2009,  Iraqi  Security  Forces  had  assumed  responsibility  for  security  in  Iraq 
under  the  Security  Agreement  and  most  of  the  18  provinces  had  completed  transition  to  Iraqi 
control.  Assessments  became  very  difficult  at  that  point  as  coalition  data  coilection  decreased 
dramaticaUy  and  Iraqi-reported  data  often  could  not  be  verified. 

77.  Cormabie,  "Embracing  the  Fog  of  War:  Assessment  and  Metrics  in  Counterinsurgency,"  p.  7. 
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largely  nascent  and  uncoordinated  across  the  theater  of  operations/®  While  a  number 
of  commands  (to  include  ISAF),  had  embedded  operations  analysts,  these  persormel 
were  largely  engaged  in  traditional  operations  research  activities  (e.g.,  analyzing  the 
effective  application  of  resources,  such  as  those  designed  to  counter  improvised 
explosive  devices).  During  one  of  the  authors’  first  visit  to  headquarters  (HQ)  ISAF  in 
October  2008,  he  met  with  the  Operational  Analysis  Group  (OAG)  and  found  that 
they  were  largely  focused  on  analyzing  quantitative  data  sources  (e.g.,  violence 
statistics)  and  developing  mathematical  models  of  the  insurgency.  There  was  no 
formal  assessment  group  or  process  at  HQ  ISAF  at  that  time. 

In  early  2009,  ISAF  established  the  Afghan  Assessment  Group  (or  AAG,  which 
absorbed  the  OAG)  to  serve  as  the  theater  lead  for  strategic  and  operational 
assessment.  The  AAG  created  a  formal  assessment  process  and  set  of  products  that 
drew  heavily  on  quantitative  metrics  and  weighted-averaging  “roITup”  techniques, 
such  as  that  described  in  Appendix  H  of  the  U.S.  Army’s  Field  Manual  5-0,  The 
Operations  Process.  This  approach  was  data-intensive,  mechanistic,  and  it  relied  on 
mathematical  formulations  that  obscured  underlying  causes  and  effects — criticisms 
that  had  already  been  levied  against  similar  approaches  used  in  Iraq  and  elsewhere. 
Additionally,  the  specific  metrics  that  the  AAG  developed  were  the  subject  of 
widespread  criticism  in  the  2009-2010  timeframe.®®  To  wit,  the  head  of  the  AAG  at 
that  time  was  quoted  as  saying,  “Our  metrics  suck.’’®' 


78.  This  section  wiU  focus  on  operations  assessment  as  practiced  at  ISAF,  in  part  because  of 
the  authors’  sustained  engagement  with  that  command.  Much  of  the  description  here  is 
adapted  from  Jonathan  Schroden,  "Operations  Assessment  at  Isaf:  Changing  Paradigms,"  in 
Innovation  in  Operations  Assessment:  Recent  Developments  in  Measuring  Results  in  Conflict 
Environments,  ed.  Andrew  P.  Williams,  et  al.  (Norfolk,  VA:  NATO  Supreme  AUied  Command 
Transformation,  2013).  The  changing  nature  of  operations  assessment  and  the  challenges  faced 
by  HQ  ISAF  were  similar  to  the  situations  at  the  Regional  Commands  (RC)  and  lower  echelons, 
as  well.  See,  for  example,  WiLLiam  Upshur,  Jonathan  Roginski,  and  David  Khcullen,  "Recognizing 
Systems  in  Afghanistan:  Lessons  Learned  and  New  Approaches  to  Operational  Assessments," 
PRISM  3,  no.  3  (June  2012).  See  also  Gerald  WoodiU,  Ewen  Stockbridge,  and  Mike  Cumberland, 
"Defence  R&D  Canada  Centre  for  Operational  Research  and  Analysis  Technical  Memorandum 
2010-008,"  in  Counterinsurgency  Campaign  Assessment  and  Direction  in  Regional  Command 
South.  Military  forces  and  civihan  organizations  faced  similar  challenges  in  assessing  the 
effects  of  their  actions;  see  Joakim  Marklund  et  al.,  "Challenges  in  Assessing  Progress  in 
Multifunctional  Operations:  Experiences  from  a  Provincial  Reconstruction  Team  in 
Afghanistan,"  in  16th  International  Command  and  Control  Research  and  Technology 
Symposium  (Quebec,  Canada,  2011). 

79.  U.S.  Department  of  the  Army,  The  Operations  Process,  field  Manual  5-0  (Washington,  DC: 
U.S.  Department  of  the  Army,  2010). 

80.  Jason  Campbell,  Michael  O'Hanlon,  and  Jacob  Shapiro,  "How  to  Measure  the  War,"  Policy 
Review  157  (2009);  Andrew  Exum  et  al.,  "Triage:  The  Next  Twelve  Months  in  Afghanistan  and 
Pakistan,"  (2009);  Jonathan  Schroden,  "Measures  for  Security  in  a  Counterinsurgency,"  Journal 
of  Strategic  Studies  32  (2009);  John  Agoglia,  Michael  Dziedzic,  and  Barbara  Sotirin,  "Measuring 
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In  mid-2009,  when  General  Stanley  McChrystal  took  command  of  ISAF,  his  personal 
critique  (which  was  leaked  to  the  press)  lent  weight  to  these  criticisms,  stating: 

ISAF  must  develop  effective  assessment  architectures... to  measure 
the  effects  of  the  strategy,  assess  progress  toward  key  objectives,  and 
make  necessary  adjustments.  ISAF  must  identify  and  refine 
appropriate  indicators  to  assess  progress,  clarifying  the  difference 
between  operational  measures  of  effectiveness  critical  to 
practitioners  on  the  ground  and  strategic  measures  more  appropriate 
to  national  capitals.®^ 

In  the  wake  of  this  conclusion,  ISAF  revised  its  assessment  processes  and  products  to 
include  both  qualitative  and  quantitative  information  in  a  more  holistic  manner.  It 
continued  to  collect  data  for,  and  to  calculate,  many  of  the  quantitative  metrics  that 
formed  the  core  of  its  previous  assessment  products,  but  it  placed  greater  emphasis 
on  the  professional  military  judgment  of  commanders  in  the  field,  especially  those  of 
the  ISAF  Joint  Command  (IJC)  and  NATO  Training  Mission-Afghanistan  (NTM-A).®^  As 
a  result  of  General  McChrystal’ s  shift  to  a  population-centric  counterinsurgency 
strategy  and  the  associated  surge  of  U.S.  and  NATO  military  forces  into  Afghanistan, 
ISAF  shifted  its  assessment  focus  to  addressing  progress  along  three  major  lines  of 
operation  (LOOs):  protecting  the  population,  building  the  Afghan  National  Security 
Forces  (ANSF),  and  increasing  the  capacity  of  the  Afghan  government.  ISAF’s 
assessment  products  also  shifted,  from  mostly  quantitative  and  data-intensive  charts 
and  graphs  to  more  nuanced  narrative  “judgments.”  General  McChrystal  also  placed 
a  heavy  emphasis  on  the  transparency  and  opermess  of  the  assessment,  pushing 
hard  to  declassify  ISAF’s  assessment  products  and  distribute  them  widely  (e.g.,  to 
academics,  think  tanks,  and  the  media).  Ultimately,  these  efforts  were  overruled  by 
his  superiors  and  ISAF’s  primary  assessment  products  remained  classified  and 
limited  in  distribution. 

The  approach  to  operations  assessment  would  change  again  in  mid-2010,  when 
General  David  Petraeus  took  command  of  ISAF  and  brought  with  him  a  stronger 
emphasis  on  transitioning  security  responsibility  to  the  Afghans.  Under  his 
command,  ISAF  rewrote  its  campaign  plan  to  include  six  LOOs,  each  with  its  own 
operational  objective  and  set  of  intermediate  objectives  (waypoints  on  the  path  to 


Progress  in  Conflict  Environments,"  (2010);  David  Kllcullen,  "Measuring  Progress  In 
Afghanistan,"  in  Counterinsurgency  (New  York:  Oxford  University  Press,  2010);  Michael 
O'Hanlon  and  Hassina  Sherjan,  "Toughing  It  out  hi  Afghanistan,"  (2010). 

Managing  Military  Organizations:  Theory  and  Practice,  ed.  Joseph  Soeters,  P.C.  van  Fenema, 
and  R.  Beeres  (New  York:  Routledge,  2010). 

Stanley  McChrystal,  "COMISAF's  Initial  Assessment,"  (2009). 

These  were  subordinate  commands  to  ISAF  that  were  newly  stood  up  around  that  time. 
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the  operational  objective).  To  assess  these  intermediate  objectives,  the  AAG  went 
back  to  a  set  of  largely  quantitative  metrics  aimed  at  “measuring”  progress  in  its 
campaign.  These  metrics  were  then  used  to  determine  where,  on  a  “thermograph”  (a 
scale  of  red  to  green),  ISAF  stood  with  respect  to  accomplishing  each  intermediate 
and  operational  objective.  ISAF’s  assessment  products  in  this  period  largely 
consisted  of  PowerPoint  shdes  showing  these  thermographs  and  specific  trends  for  a 
variety  of  quantitative  indicators  (e.g.,  level  of  violence)— all  of  which  were  classified. 

General  John  R.  Allen,  who  took  command  of  ISAF  in  September  2011,  made  even 
more  fundamental  changes  to  assessment  by  directing  the  AAG  to  redesign  the 
process  altogether.  Dissatisfied  with  the  heavy  emphasis  on  quantitative  metrics  and 
lack  of  context  that  he  observed  in  the  command’s  assessment  products,  he  directed 
the  AAG  to  be  “revolutionary”  in  creating  something  new.  He  also  directed  the  AAG 
to  reach  out  to  a  number  of  experts  in  the  operations  assessment  field  to  solicit  their 
assistance.**"* 

The  new  assessment  process  and  products  created  by  the  AAG  did  not  conform  to 
either  U.S.  or  NATO  doctrine;  rather,  the  new  assessment  “paradigm”  was  designed 
specifically  to  address  the  unique  complexities  of  the  counterinsurgency  in 
Afghanistan  and  the  needs  of  the  ISAF  Commander  at  that  time.***  The  assessment 
paradigm  had  a  two-tier  structure  consisting  of  both  strategic  and  campaign 
assessments.  The  strategic  assessment  focused  on  answering  so-called  “Strategic 
Questions”— high-level,  multifaceted  questions  designed  to  be  of  primary  interest  to 
senior  decision-makers— in  narrative,  analytic  form.  The  operational  assessment  used 
sets  of  qualitative  standards  and  accompanying  narrative  responses  to  gauge 
accomplishment  of  campaign  tasks.  Both  tiers  were  designed  to  capture  the  current 
state  of  the  war  while  emphasizing  a  need  to  anticipate  and  prepare  for  future 
challenges  and  opportunities.  Additionally,  the  two  assessments  and  their  associated 
processes  were  designed  to  stimulate  discussions  leading  directly  to  decisions  by 
senior  leaders  on  actions  they  could  take,  direct,  or  request.  In  essence,  this 
approach  provided  them  the  information  and  impetus  to  make  (or  argue  for)  changes 
and  actions  critical  to  the  continued  progress  of  the  campaign. 

ISAF  used  this  two-tier  assessment  system  for  nearly  a  year,  with  continued 
modifications  to  the  Strategic  Questions  and  specific  operational  standards.  In  mid- 
2013,  the  AAG  dropped  the  operational  assessment  component  and  .streamlined  the 
assessment  development  process  by  reducing  the  number  of  coordination  meetings 


84.  FuU  disclosure:  one  of  the  authors  (Schroden)  was  asked  to  deploy  to  help  the  AAG  with 
this  redesign  and  did  so  from  November  2011  to  February  2012. 

85.  The  assessment  paradigm  developed  by  the  AAG  in  late  2011/early  2012  is  described  in 
detail  in  Schroden,  "Operations  Assessment  at  ISAF:  Changing  Paradigms."  See  also  "A  New 
Paradigm  for  Assessment  in  Counterinsurgency,"  Military  Operations  Research  18,  no.  3  (2013). 
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and  assessment  working  groups.  As  of  the  writing  of  this  paper,  ISAF  was  still  using 
the  Strategic  Questions  and  continued  to  structure  its  main  assessment  report  (the 
Quarterly  Strategic  Assessment  Report,  or  QSAR)  around  them,  though  it  also 
retained  armexes  to  the  report  containing  significant  amounts  of  quantitative 
information  (e.g.,  violence  statistics  and  polling  results). 


Implic  ations  for  Opeiations  Assessment 


The  wars  in  Iraq  and  Afghanistan  witnessed  the  collision  of  vague  political  objectives 
with  complex,  ill-defined  battlefields.  At  the  same  time,  the  overly  reductionist 
systems  thinking  present  in  the  Vietnam  era  reemerged,  coupled  with  Information 
Age  increases  in  computing  speed  and  the  ability  to  collect,  store,  and  analyze  large 
volumes  of  data.  As  a  result,  analysts  and  assessors  often  found  themselves  awash  in 
data,  but  unable  to  convert  them  into  information  useful  to  senior  decision  makers, 
who  were  often  struggling  to  grasp  the  situation  on  the  ground.  By  2010,  it  was  not 
uncommon  to  hear  the  environment  in  theater  described  as  “data  rich,  but 
knowledge  poor.” 

Despite  the  problems  that  assessment  teams  encountered  in  these  two 
counterinsurgencies,  there  were  exceptions  that  showed  that  the  use  of  quantitative 
data  in  building  assessments  is  not  entirely  misguided.  In  Iraq,  for  example, 
problems  resulted  from  the  application  of  assessment  methods  without  considering 
the  complex  nature  of  the  war,  and  from  the  effort  to  centralize  results  in  what  was  a 
highly  localized  conflict.  However,  some  analysts  (often  working  at  operational  and 
tactical  levels),  were  able  to  mitigate  these  issues.  For  example,  one  of  the  authors 
(Schroden)  worked  with  MNF-W  to  develop  quantitative  metrics  and  analyses  tied 
directly  to  specific  theories  of  counterinsurgency  and  objectives  derived  from  them.'*'® 
These  metrics  suffered  from  some  of  the  same  difficulties  as  other  quantitative 
approaches  (e.g.,  imperfect  data  sources),  but  because  they  were  calculated  at  a  lower 
echelon  of  command  and  were  related  directly  to  hypotheses  about  how  to 


86.  "Measures  for  Security  in  a  Counterinsurgency."  One  metric  that  was  deemed  particularly 
useful  in  demonstrating  progress  in  al  Anbar  was  the  ratio  of  friendly-initiated  to  enemy- 
initiated  attacks.  This  ratio  is  tied  directly  to  a  theory  of  counterinsurgency  that  says  initiative 
is  one  of  the  few  advantages  that  an  insurgent  force  can  have  over  a  typically  stronger,  better 
equipped  counterinsurgent  force.  As  such,  knowing  whether  the  counterinsurgent  force  is 
reducing  this  advantage  of  initiative  is  fundamental  to  knowing  whether  it  is  being  successful. 
Another  example  of  a  useful  metric  was  the  number  of  tips  that  Iraqi  civilians  provided  to 
coalition  forces  to  aid  in  countering  the  enemy.  While  insufficient  to  provide  a  sweeping 
assessment  on  its  own,  this  metric  reflected  many  Important  conditions  on  the  ground,  such  as 
local  support  for  counterinsurgency  efforts  and  locals'  sense  of  security  as  reflected  in  their 
willingness  to  actively  risk  retribution  by  reporting  insurgent  activities. 
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effectively  prosecute  the  counterinsurgency  campaign  in  al  Anbar,  they  were 
nonetheless  deemed  useful  by  decision-makers  at  the  command. 

Of  note  for  the  war  in  Iraq  and  especially  in  Afghanistan,  commanders  have  had  a 
notable  impact  on  the  assessments  process,  and  their  shifting  preferences  have  often 
directly  generated  the  oscillation  between  quantitative  and  qualitative  approaches.  In 
Iraq,  operations  assessment  was  a  central  issue  for  a  relatively  short  period  of  time. 
The  assessments  process  underwent  some  changes  with  new  MNF-I  commanders,  but 
as  violence  levels  began  to  drop  dramatically  in  2007,  so  did  the  demand  for 
assessments  and  the  perceived  need  to  make  significant  changes  to  the  assessment 
process.  Essentially,  commanders  had  tangible  results,  which  satisfied  interested 
parties.®^  The  war  in  Afghanistan,  in  contrast,  has  lasted  much  longer  and 
commanders  have  taken  highly  disparate  approaches  to  operations  assessment. 
Because  progress  has  been  difficult  to  demonstrate,  ISAF  commanders  have  sought 
out  new  ways  to  understand  the  environment  and  to  communicate  their 
understanding  to  policy  leaders. 

These  persistent  changes  to  the  assessments  process  in  part  reflect  the  commander’s 
prerogative,  as  outlined  in  current  and  past  doctrine,  since  the  burden  of  proof  to 
show  progress  rests  largely  on  his  shoulders.  But  in  a  data-rich  age  where  an 
audience  is  accustomed  to  seeing  numbers  to  back  up  opinions  or  claims,  a 
commander’s  judgment  has  proved  a  necessary  but  insufficient  component  of  the 
assessments  process.  While  EBA  ostensibly  provided  a  framework  to  bring 
quantitative  corroboration  to  qualitative  judgments,  its  manifold  and  consistent 
inadequacies  have  prompted  commanders  to  seek  alternatives.  The  doctrinal  vacuum 
that  followed  the  death  of  EBO/EBA  and  the  malleable  nature  of  the  assessments 
processes  that  have  been  created  in  EBA’s  wake  have  allowed  for  significant  variation 
in  the  way  commanders  approach  assessment.  Some  have  considered  it  an  activity 
designed  to  produce  and  analyze  precise  measurements  of  on-the-ground 
performance.  Others  have  leveraged  assessment  capabilities  as  one  component  of  a 
broader  messaging  effort  to  communicate  and  further  progress  towards  their 
objectives.  Regardless  of  the  approach,  each  commander  has  sought  effective  ways  to 
produce  assessments  that  can  stand  up  under  a  high  level  of  scrutiny  from  top 
defense  and  policy  leaders. 

We  are  still  in  the  midst  of  what  General  Mattis  called  the  “renaissance”  in  operations 
assessment.  Its  rare  successes  and  common  failures  in  Iraq  and  Afghanistan  have 
elevated  the  discussion  of  the  issue  to  unprecedented  levels  in  U.S.  military  history. 
In  recent  years,  a  number  of  people  with  direct  experience  in  conducting  or 
supporting  operations  assessment  in  both  theaters,  along  with  an  increasing  number 


87.  There  were  certainly  changes  to  the  assessments  process  at  lower  echelons  (e.g.,  MNF-W), 
but  there  were  not  the  major  overhauls  at  the  highest  levels  that  we  have  seen  in  Afghanistan. 
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of  academic  researchers,  have  begun  publishing  hypotheses  for  why  assessments 
failed  to  live  up  to  their  theoretical  promise  in  Iraq  and  Afghanistan.  For  example, 
Ben  Connable  has  examined  in  detail  the  parallels  and  pitfalls  of  the  systems 
thinking  and  quantitative  metrics  approaches  used  in  Vietnam,  Iraq,  and  Afghanistan 
and  argued  that  an  emphasis  on  quantitative  analysis  for  gauging  progress  in 
counterinsurgency  is  fundamentally  unsound  given  its  nature  as  a  social  (versus 
nation-state)  conflict.®*  Others  such  as  Stephen  Downes-Martin  have  argued  that  the 
reductionist  mathematical  approach  taken  by  U.S.  and  NATO  doctrine  (most  notably 
in  the  effects-based  approach  to  operations  and  assessment)  is  flawed  in  that  it 
greatly  oversimplifies  the  complexities  of  counterinsurgency  warfare  and  assumes  a 
much  higher  level  of  capability  for  modeling  and  analyzing  it  than  we  currently 
possess.*®  Jonathan  Schroden  has  argued  that  in  addition  to  these  flaws  in  theory, 
doctrine,  and  practice,  the  military  has  failed  to  educate,  train,  and  properly  prepare 
those  being  tasked  to  conduct  and  consume  operations  assessment  in  the  field.®® 
Other  authors  continue  to  press  the  notion  that  the  problem  remains  one  of  not 
having  the  right  quantitative  metrics.®^ 

These  authors  have  clearly  not  yet  reached  a  consensus  answer  to  the  question  of 
why  operations  assessments  in  Iraq  and  Afghanistan  were  so  challenging  and  often 
failed  to  provide  the  kinds  of  insights  senior  decision-makers  (and  the  general 
public)  desired.  However,  it  is  clear  that  there  exists  widespread  dissatisfaction  with 
operations  assessment  doctrine  and  the  assessment  principles  that  accompanied 
concepts  like  effects-based  operations.  This  dissatisfaction  came  to  a  head  at  a 
special  meeting  held  jointly  by  the  Military  Operations  Research  Society  (MORS)  and 
the  U.S.  Central  Command  (CENTCOM)  in  November  2012.®®  This  meeting  consisted 
of  plenary  sessions  and  working  groups  focused  on  strategic  theater,  campaign, 
operational,  and  tactical  assessments  (among  other  topics).  One  of  the  authors  was  a 
presenter  at  the  meeting,  and  noted  that  the  level  of  frustration  with  the  current 
state  of  operations  assessment  among  attendees  was  palpable  and  surprisingly 
consistent  across  many  of  the  working  groups.  It  seemed  that  no  matter  what  level  of 
command  the  attendees  had  experience  in,  or  whether  their  experiences  were  in  Iraq, 
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Afghanistan,  or  other  regions  of  the  world,  they  encountered  similar  levels  of 
frustration  trying  to  design  and  implement  effective  operations  assessment 
processes.  It  appears,  therefore,  that  consensus  exists  on  at  least  one  level:  as 
Downes-Martin  asserted,  “operations  assessment  ...  is  broken.” 
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In  examining  the  history  of  operations  assessment,  we  find  consistent  themes  in  its 
evolution.  We  will  first  discuss  these  important  themes,  and  then  briefly  comment  on 
the  present  state  and  possible  future  challenges  for  operations  assessment. 


historical  IhemesofOpeiations  Assessment 

Oscillation  between  quantitative  and  qualitative 
approaches 

One  of  the  most  readily  apparent  themes  in  operations  assessment  history  has  been 
the  oscillation  between  highly  quantitative  approaches  and  more  balanced 
approaches  that  include  quantitative  inputs,  but  place  significant  value  on  qualitative 
considerations.  Those  responsible  for  operations  assessment  have  struggled  to 
produce  assessments  that  exhibit  both  precision  and  insight,  often  forsaking  one  for 
the  other.®  Myriad  pressures  have  exacerbated  the  challenges  associated  with  finding 
this  balance,  including  the  persistent  demand  for  progress  reports  from  internal  and 
external  audiences,  the  compulsion  to  leverage  new  technologies  to  improve  the 
assessments  process,  and  the  lack  of  a  proven  assessments  framework  for  regular 
and  irregular  warfare. 

In  the  Vietnam  era,  the  combination  of  a  heavy  emphasis  on  systems  analysis  and  the 
uncertainty  and  inconsistency  that  characterizes  a  counterinsurgency  produced  an 
unfortunate  confluence.  In  that  environment,  centralized  assessments  became 
singularly  focused  on  quantifying  the  unquantifiable  in  the  effort  to  measure 
progress.  The  general  lack  of  qualitative  inputs  excluded  important  contextual 
information  and  weakened  the  overall  assessment.  The  subsequent  backlash  against 
the  methods  used  in  Vietnam  assessments  coincided  with  a  period  when  the  United 
States  was  committed  to  deterring  war,  rather  than  fighting  it.  This  status  obviated 
any  perceived  need  to  fix  the  problems  assessors  encountered  in  Vietnam  or  produce 
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a  better  method  for  assessing  operations.  Consequently,  the  Cold  War  saw  operations 
assessment  return  to  its  pre-Vietnam  status  almost  by  default.  Quantitative  inputs 
like  battle  damage  assessment  were  combined  with  qualitative  factors— most 
importantly  the  commander’s  judgment— to  form  an  overall  assessment. 

The  Information  Age  brought  renewed  emphasis  on  quantitative  methods  with  the 
idea  that  advanced  computing  capabilities  could  make  possible  the  level  of  precision 
that  had  eluded  assessors  in  Vietnam.  The  transformation  occurring  in  other  sectors 
like  business  and  finance  only  reinforced  the  adamancy  in  military  circles  that  this 
“revolution”  would  make  efforts  like  operations  assessment  a  mere  matter  of 
leveraging  the  right  technical  assets  and  mathematical  models.  Embodied  in  the 
effects-based  assessment  framework,  this  line  of  thinking  dominated  the 
assessments  process  for  much  of  the  wars  in  Iraq  and  Afghanistan,  despite  the  fact 
that  pitfalls  and  obstacles  associated  with  EBA  resulted  in  highly  skewed  progress 
reports. 

Emphasison  operations  assessment  is  proportional  to 
the  lackof  obvious  progress 

A  brief  study  of  the  history  of  operations  assessment  also  reveals  that  the  level  of 
effort  dedicated  to  the  process  (including  efforts  to  change  or  improve  it)  tends  to  be 
proportional  to  the  level  of  demand  for  assessments.  In  turn,  the  demand  for 
assessments  seems  to  be  highest  when  our  political  objectives  are  murkiest  or 
strategic  models  are  weakest.  When  progress  is  readily  apparent  and  the  traditional 
approaches  of  battle  damage  assessment  and  the  commander’s  estimate  satisfy,  the 
operations  assessment  process  remains  fairly  static  and  receives  little  attention.  If  a 
commander  can  observe  developments  in  real  time  and  communicate  up  to  higher 
echelons  with  tangible  results  (e.g.,  ground  taken,  targets  destroyed,  or  enemy  killed), 
demand  for  assessments  is  low.”'*  However,  m  more  protracted  conflicts — especially 
those  that  are  shrouded  in  the  ambiguities  of  irregular  warfare— objectives  tend  to  be 
opaque  and  shifting,  so  leaders  work  to  find  ways  to  communicate  progress,  which 
may  not  be  readily  apparent  and  is  often  difficult  to  measure.  This  dynamic  was  first 
evident  m  Vietnam,  where  operations  assessment  gained  significant  attention  as 
signs  of  progress  proved  elusive.  The  quantitative  methods  that  came  to  dominate 
the  assessments  process  were  so  pervasive,  in  part,  because  they  were  fairly  easy  to 
comprehend  and  came  at  a  time  when  U.S.  leadership  was  clamoring  for  evidence 
that  efforts  were  succeeding.  Daddis  summarizes  this  dynamic  in  his  treatment  of 


94.  James  G.  Diehl  and  Charles  E.  Sloan,  "Battle  Damage  Assessment:  The  Ground  Tmth,"  Joint 
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the  assessments  process  in  Vietnam  by  saying  that  “numbers  whetted  an  appetite  for 
more  numbers.”®^  Supply  rose  to  meet  demand,  often  at  the  expense  of  accuracy. 

In  contrast,  there  was  little  demand  for  robust  assessments  in  the  decades  that 
followed  the  Vietnam  War  because  U.S.  Cold  War  policy  concentrated  on  comparisons 
of  capabilities  in  the  absence  of  open  conflict.  Doctrine  on  operations  assessment, 
which  was  relatively  sparse  and  nested  in  the  planning  process,  provided  as  much 
guidance  as  was  deemed  necessary.  As  the  United  States  became  much  more  globally 
engaged  following  the  collapse  of  the  Soviet  Union,  however,  the  need  for  a  useful 
way  to  measure  progress  in  nonconventional  operations  reemerged.  While  this 
changing  dynamic  was  evident  throughout  the  1990s,  it  took  on  much  more 
significance  during  the  counterinsurgency  operations  in  Iraq  and  Afghanistan.  Once 
again,  the  lack  of  clear  progress  in  the  midst  of  prolonged  conflict  drove  demand  for 
more  rigorous  (or  simply  better)  ways  to  measure  the  impact  of  tens  of  thousands  of 
boots  on  the  ground.  Initially,  leaders  assumed  EBA  would  meet  that  need,  but  its 
dismissal  left  a  doctrinal  vacuum  that  remains  unfilled.  With  doctrine  falling  short 
amidst  persistent  calls  for  signs  of  progress,  there  was,  and  continues  to  be, 
significant  support  for  a  thorough  revamp  of  the  operations  assessment  process. 

Operations  assessment  has  to  serve  distinct,  and 
competing,  audiences 

The  demand  for  operations  assessments  typically  originates  with  two  distinct 
audiences  that  drive  the  process  in  very  deliberate,  but  often  competing,  directions. 
The  first  audience  is  internal  to  the  unit:  the  commander,  staff,  and  other  plarmers. 
Assessments  for  this  audience  serve  the  purpose  of  informing  the  planning  process 
as  dictated  by  doctrine,  moving  “down  and  in”  for  use  inside  the  force.®®  As  such, 
often  there  is  a  degree  of  skepticism  of  assessment  inputs  and  results,  since  the 
commander  and  his  staff  seek  the  most  accurate  and  objective  material  to  form  the 
basis  from  which  they  plan  future  operations.  There  can  be  a  greater  appetite  for 
raw,  unaltered  data  to  help  build  the  fullest  understanding  of  the  battlespace.  The 
internal  nature  of  the  planning  process  allows  a  commander  to  work  with  such 
unpolished  assessments  without  significant  scrutiny. 

By  contrast,  policymakers  and  other  senior  leadership  comprise  the  second  audience 
and  they  have  brought  a  high  level  of  scrutiny  to  the  assessments  process  in  recent 
years.  Assessments  inform  their  understanding  and  perception  of  progress,  which 
aids  in  their  determination  of  whether  to  support  a  commander’s  recommendations 

95.  Daddis,  No  Sure  Victory:  Measuring  U.S.  Army  Effecttceness  and  Progress  in  the  Vietnam 
War,  p.  53. 
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36 


CNA 


and  ultimately  decides  the  duration  and  extent  of  U.S.  commitment  in  a  conflict.  This 
dynamic  has  produced  a  propensity  in  operations  assessment  that  is  worth 
examining.  Assessments  that  reflect  bottom-up  data  collection  and  analysis,  while 
ostensibly  objective,  risk  creating  dissatisfaction  among  higher  leadership  if  the 
results  do  not  show  sufficient  progress.  As  a  consequence,  unlike  the  skeptical 
approach  commanders  often  take  to  assessments  in  the  planning  process,  those 
delivered  “up  and  out”  to  pohcymakers  often  reflect  an  optimism  that  is  either  not 
present  in  or  even  counter  to  internal,  bottom-up  assessments. 

In  the  wars  in  Iraq  and  Afghanistan,  we  have  observed  some  high-level  commanders 
using  assessments  as  a  vehicle  to  deliver  a  specific  message  to  the  policymaking 
audience,  adding  a  new  dimension  to  the  evolution  of  operations  assessment.  To  a 
large  extent,  this  tendency  reflects  the  commander's  response  to  pressure  generated 
by  the  existence  of  two  distinct  audiences.  Both  groups  demand  his  attention,  but  the 
assessments  for  each  are  often  built  with  different  inputs  and  approaches.  When  a 
commander  focuses  more  on  one  audience  over  another,  it  directly  affects 
assessment  methods  and  approaches.  What  a  commander  chooses  to  measure 
reveals  his  priorities,  and  when  civilian  leadership  is  the  primary  audience,  that 
selection  tends  to  shift  to  include  those  metrics  that  can  best  bolster  the  position  he 
intends  to  defend.  This  influence  threatens  to  erode  the  objectivity  of  the 
assessment,  but  mitigates  the  risk  of  losing  support  from  higher  leadership. 
Ultimately,  the  effort  to  satisfy  very  different  audiences  has  hindered  the 
development  of  an  effective  framework  for  operations  assessment,  as  the  processes 
used  have  changed  significantly  depending  on  the  commander  and  the  audience. 

Ope  ratio  ns  assessment  tends  to  follow  trends  in  civilian 
and  business  sectors 

Another  notable  trend  in  operations  assessment  history  is  the  tendency  for  its 
development  to  mirror  major  trends  in  civilian  circles.  One  of  the  clearest 
manifestations  of  this  phenomenon  is  in  the  parallels  between  the  development  of 
operations  assessment  and  business  management  practices.  The  influence  of 
business  on  operations  assessment  first  became  evident  in  World  War  II,  as  the 
assessment  effort  came  to  include  business-oriented  minds  like  Robert  McNamara. 
This  influence  on  operations  assessment  grew  exponentially  during  McNamara’s  time 
at  the  Pentagon,  where  he  applied  his  systems  analysis  approach  to  the  Vietnam 
assessments  process.  Years  later,  the  growing  popularity  of  results-based 
management  from  the  late  1980s  through  the  Information  Age  found  its  way  into 
operations  assessment,  most  clearly  in  effects-based  operations  doctrine.  However, 
like  McNamara  and  systems  analysis,  this  translation  has  revealed  some  of  the 
limitations  in  applying  business  practices  to  military  operations  assessments.  Even 
with  the  same  technology  that  is  revolutionizing  the  way  the  world  uses  data  to 
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measure  results  in  business,  the  operations  assessment  process  is  still  hindered  by 
the  lack  of  a  useful  framework  for  exploiting  vast  amounts  of  data. 


Rjtuie  C  hallenges  of  Opemtions  Assessment 


Having  an  understanding  of  the  history  of  operations  assessment  allows  us  to  look 
ahead  and  make  some  informed  speculation  as  to  what  challenges  may  face 
practitioners  in  the  future. 

Meeting  consumerand  audience  expectations 

Tracing  the  patterns  of  U.S.  business  development  may  provide  some  indication  of 
where  operations  assessment  is  headed,  and  how  it  can  be  shaped  so  that  it  benefits 
from  successful  business  practices  and  avoids  repeating  past  missteps.  For  example, 
business  trends  like  customer  centricity  wiU  likely  only  add  to  the  demand  for 
accurate  and  timely  assessments,  since  consumers  are  increasingly  accustomed  to 
receiving  goods  and  services  that  are  tailored  to  the  individual  and  delivered  rapidly. 
New  technologies  have  enabled  this  development,  freeing  up  human  effort  to  focus 
on  ensuring  customer  satisfaction.  Although  the  defense  industry  has  access  to 
similar  technologies,  they  have  encountered  significant  obstacles  in  trying  to 
automate  certain  processes,  including  assessments.  Therefore,  meeting  expectations 
of  modern  audiences  without  compromising  the  integrity  of  the  assessment  may 
prove  even  more  challenging  in  the  future. 

Increasing  emphasison  operations  assessment,  but 
continuing  discord  on  how  it  should  be  conducted 

As  the  United  States’  commitment  in  Afghanistan  diminishes,  it  is  worthwhile  to 
consider  how  the  future  operating  environment  might  affect  operations  assessment. 
The  current  momentum  behind  doctrinal  revision  is  promising,  but  demand  for 
assessments  may  dwindle  as  forces  have  fewer  operations  to  assess.  Because  there  is 
a  tendency  to  neglect  assessments  when  there  is  not  significant  demand  for  them, 
the  potential  for  a  return  to  obscurity  exists.  However,  two  important  dynamics  point 
to  a  likely  break  with  this  historical  trend.  First,  as  discussed  above,  there  is  broad 
agreement  that  the  development  of  a  sound  assessments  process  is  a  valuable 
pursuit,  and  many  defense  organizations  have  dedicated  resources  to  that  effort. 
Secondly,  according  to  national  strategic  policy,  the  United  States  will  continue  to  be 
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engaged  around  the  globe  in  limited  footprint  operations  in  the  coming  years. In  the 
post-Vietnam  era  the  United  States  was  singularly  focused  on  deterring  a  peer 
competitor  and  its  global  engagement  was  comparatively  minimal.  In  that  climate, 
assessments  played  an  insignificant  role.  That  bipolar  dynamic  no  longer  exists,  and 
likely  will  not  exist  in  the  near  future.  Indeed,  the  new  security  environment  is 
actually  contributing  to  the  momentum  behind  operations  assessment. 

Today,  the  United  States  engages  in  operations  ranging  from  theater  security 
cooperation  to  crisis  response  on  a  regular  basis.  Civilian  government  and  non-profit 
organizations  have  built  significant  experience  in  the  discipline  of  evaluation^*  of 
efforts  like  development  and  reconstruction,  which  has  had  notable  influence  on 
military  operations  assessment  and  helped  propel  the  effort  to  build  an  effective 
process.  Additionally,  policymakers,  who  are  now  accustomed  to  receiving  frequent 
progress  reports  on  military  engagements,  will  likely  continue  to  expect  assessments 
from  “Phase  0”  operations  going  forward.  There  are  current  examples  that  affirm  this 
trend,  such  as  the  requirement  for  the  geographic  combatant  commands  (GCC)  to 
produce  a  theater  campaign  plan  and  assess  against  that  plan  on  an  annual  basis. 
Requiring  such  reports  from  permanent,  standing  units  points  to  the  significance  of 
operations  assessment  in  the  modern  context  and  makes  it  less  likely  that  it  will  fade 
into  obscurity  going  forward.  However,  even  with  this  continued  emphasis,  there  are 
current  and  emerging  dynamics  that  add  to  the  uncertainty  surrounding  exactly  how 
operations  assessment  will  develop. 

The  challenges  associated  with  the  assessments  process  in  Iraq  and  Afghanistan  have 
elevated  operations  assessment  to  a  level  unprecedented  in  U.S.  military  history.  As 
the  United  States  moves  forward  from  these  conflicts,  there  is  broad  consensus  that 
the  assessments  process  is  broken;  however,  there  is  not  a  clear  way  forward  on  how 
to  fix  it.  The  vacuum  left  in  the  wake  of  EBA  remains,  though  various  bodies  are 
dedicating  significant  effort  to  fill  it.  The  overhaul  of  assessments  doctrine  is  already 
occurring  in  several  places.  The  Air  Land  Sea  Application  Center  has  produced  a 
multiservice  tactics,  techniques,  and  procedures  (MTTP)  publication  on  operations 
assessment  that  lists  the  basic  tenets  of  the  process  and  provides  examples  of 
approaches  that  have  been  useful  in  the  past.  The  most  influential  doctrinal 
publication  on  irregular  warfare  to  come  out  in  decades.  Army  Field  Manual  3-24  on 
Counterinsurgency,  provided  almost  no  guidance  on  operations  assessment  in  its 
first  iteration  in  2006.  It  has  since  been  updated  and  the  current  version  of  EM  3-24 
dedicates  a  chapter  to  assessments  and  includes  much  greater  depth  on  the  issue. 


97.  See,  for  example.  White  House,  "National  Security  Strategy,"  (2010). 

98.  The  civiUan  sector  typically  uses  the  word  “evaluation”  to  refer  to  measuring  progress, 
rather  than  assessment.  The  experience  in  and  Uterature  on  evaluation  in  civilian  circles  is 
extensive.  While  we  do  not  address  it  in  depth  here,  evaluation  in  non-military  applications  is 
important  to  consider  when  thinking  about  the  future  of  operations  assessment. 
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The  same  is  true  of  the  joint  publication  on  counterinsurgency  (JP  3-24),  which  was 
updated  in  November  2013.  There  are  also  plarmed  efforts  to  revise  joint  doctrine  on 
assessments  (specifically,  the  sections  of  JP  3-0  and  JP  5-0  that  discuss  the  topic). 
However,  these  publications  still  fall  short  in  providing  consensus  on  the  future  of 
operations  assessment  and  the  necessary  practical  guidance  on  the  actual  process  of 
conducting  assessments.  Doctrinal  revisions  would  benefit  from  the  inclusion  of 
lessons  learned  in  operations  assessment  over  the  past  several  decades,  but  it  is  also 
important  to  be  forward-looking  to  help  equip  leaders  and  assessors  and  avoid  cold 
starts  in  future  conflicts. 

Continuing  lack  of  training  and  education  on  how  to 
conduct  operations  assessment 

The  consensus  that  has  emerged  regarding  the  need  to  “fix”  operations  assessment  is 
helpful  and  has  led  to  significant  efforts  to  revise  assessment  doctrine.  However, 
there  appear  to  be  no  concurrent  efforts  within  the  U.S.  military  to  address  the 
general  lack  of  training  and  education  provided  to  those  tasked  with  conducting 
operations  assessment.  Over  the  past  decade,  the  ORSA  community  has  had  the  task 
of  operations  assessment  fall  to  them,  and  they  have  begrudgingly  accepted  and 
conducted  the  mission— often  to  the  detriment  of  their  ability  to  support 
commanders  and  units  with  traditional  operations  research  activities.  Yet,  there 
remain  significant  debates  about  whether  ORSA  techniques  are  the  most  optimal 
ones  for  assessing  progress  in  military  operations.  Beyond  simply  assuming  that 
ORSAs  inherently  have  the  best  skill  set  for  operations  assessment,  it  would  behoove 
the  military  to  think  about,  research,  and  experiment  with  other  skill  sets  and  to  use 
the  results  of  such  studies  and  experiments  to  inform  the  creation  of  training  and 
educational  courses  for  those  expected  to  conduct  operations  assessment  in  the 
future.  Even  if  assessments  doctrine  gets  “fixed,”  simply  handing  it  to  ORSAs  or 
untrained  staff  officers  does  not  fairly  empower  them  to  be  successful  in  the 
challenging  endeavor  that  is  operations  assessment. 

The  rise  of  “big  data"  amidst  limited  analytical 
resources 

The  rise  of  “big  data”  and  social  networking  promises  to  have  a  profound  impact  on 
the  future  of  operations  assessment.  As  private  and  public  sectors  endeavor  to 
organize  and  leverage  the  vast  amount  of  data  that  is  produced  and  collected  in 
short  periods  of  time,  it  is  important  to  consider  the  associated  risks  and 
opportunities  for  operations  assessment.  The  challenge  of  too  much  data  is  not  new 
to  military  assessments  and  has  plagued  the  process  since  Vietnam.  However,  there 
have  been  exponential  increases  in  the  amount  of  available  data  since  that  time,  and 
the  pace  of  that  growth  is  only  getting  faster.  This  “Information  Age  2.0”  is  already 
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having  a  substantial  impact  across  many  sectors.  Today,  the  collection  capabilities  of 
military  assets  are  phenomenal,  but  the  capacity  to  analyze  collected  data  is  severely 
lacking.  In  operations  assessment,  we  risk  reverting  back  to  an  EBO-like  mindset, 
where  the  magnitude  of  the  data  behes  the  challenges  associated  with  understanding 
and  leveraging  it.  Defense  and  business  leaders  have  already  dedicated  significant 
resources  to  finding  ways  and  building  tools  to  deal  with  big  data,  which  may  prove 
extremely  beneficial.  However,  approaching  the  use  of  big  data  in  assessments  with 
caution  and  bearing  in  mind  the  lessons  of  the  past  can  assist  in  enhancing  the 
integrity  and  reliability  of  the  process. 

The  changing  conduct,  but  steadfast  nature,  of 
warfare 

Military  theorists  and  futurists  continue  to  pontificate  as  to  how  warfare  will  be 
conducted  in  the  future.  Concepts  such  as  “hybrid  warfare”  and  “fourth  generation 
warfare”  have  been  developed  and  are  useful  constructs  for  thinking  about  what 
types  of  operations  we  may  be  asked  to  assess  going  forward.  However,  students  of 
military  history  know  that  the  underlying  nature  of  warfare  remains  constant 
through  the  ages.  War  is  uncertain  and  often  ambiguous,  and  operations  assessment 
will  continue  to  be  challenging  as  a  result.  Years  of  experience  in  assessment  and  a 
determination  to  institutionalize  the  lessons  learned  as  a  result  of  this  history  have 
the  potential  to  leave  us  better  postured  for  future  challenges  in  assessing  progress. 
As  the  operating  environment  shifts  and  U.S.  commitments  change,  the  conduct  of 
operations  assessment  will  likely  need  to  change  as  well.  But  having  reliable 
assessment  methods  to  enable  better  planning  and  decision  making  at  all  levels  is 
likely  to  be  a  constant  requirement  going  forward.  Therefore,  we  must  harness  the 
current  momentum  in  the  operations  assessment  community  to  address  the  likely 
challenges  of  future  assessment  and  ensure  the  development  of  effective  operations 
assessment  processes  going  forward. 


41 


Bibliogiaphy 


CNA 


Agoglia,  John,  Michael  Dziedzic,  and  Barbara  Sotirin.  "Measuring  Progress  in  Conflict 
Environments."  2010. 

Bracken,  Paul.  "Net  Assessment:  A  Practical  Guide."  Parameters  36,  no.  1  (Spring  2006). 

Brigham,  Erwin  R.  "Pacification  Measurement  in  Vietnam:  The  Hamlet  Evaluation  System." 
In  SEATO  Internal  Security  Seminar.  Manila,  1968. 

Campbell,  Jason,  Michael  O'Hanlon,  and  Jacob  Shapiro.  "How  to  Measure  the  War."  Policy 
Review  157  (2009):  15-30. 

Cebrowski,  Arthur  K.  "Network-Centric  Warfare."  Military  Technology  27,  no.  5  (2003):  16- 
22. 

- .  "Network-Centric  Warfare:  Its  Origin  and  Euture."  U.S.  Naval  Proceedings  124 

(January  1998). 

Center  for  Strategic  and  International  Studies.  "Progress  or  Peril?  Measuring  Iraq's 
Reconstruction."  September  2004. 

Clancy,  James,  and  Chuck  Crossett.  "Measuring  Effectiveness  in  Irregular  Warfare." 
Parameters  37,  no.  2  (Summer  2007):  88-100. 

Cohen,  Craig.  "Measuring  Progress  in  Stabilization  and  Reconstruction."  In  Stabilization 
and  Reconstruction  Series  No.  1:  USIP,  March  2006. 

Cohen,  Eliot  A.  "Net  Assessment:  An  American  Approach."  April  1990. 

Connable,  Ben.  "Embracing  the  Eog  of  War:  Assessment  and  Metrics  in 
Counterinsurgency."  2012. 

Cordesman,  Anthony.  "The  Uncertain  "Metrics"  of  Afghanistan  (and  Iraq)."  May  2007. 

Daddis,  Gregory  A.  No  Sure  Victory:  Measuring  U.S.  Army  Effectiveness  and  Progress  in  the 
Vietnam  War.  New  York,  NY:  Oxford  University  Press,  2011. 

Diehl,  James  G.,  and  Charles  E.  Sloan.  "Battle  Damage  Assessment:  The  Ground  Truth." 
Joint  Porces  Quarterly  37  (Spring  2005):  59-64. 

Downes-Martin,  Stephen.  "Operations  Assessment  in  Afghanistan  Is  Broken:  What  Is  to  Be 
Done?".  Naval  War  College  Review  64,  no.  4  (Autumn  2011):  103-25. 


42 


CNA 


Druker,  Peter  F.  Management:  Tasks,  Responsibilities,  Practices.  New  York:  Truman  Talley 
Books,  1986. 

Exum,  Andrew,  Nathaniel  Pick,  Ahmed  Humayun,  and  David  Kilcullen.  "Triage:  The  Next 
Twelve  Months  in  Afghanistan  and  Pakistan."  2009. 

Flemming,  Stephan  B.,  and  WilUam  J.  Owen.  Measurement  of  Post-Conflict  Resolution  in 
Bosnia-Herzegovina.  The  Cornwallis  Group  Vi:  Analysis  for  Assessment, 
Evaluation  and  Crisis  Management,  edited  hy  Alexander  E.  R.  Woodcock  and  David 
E.  Davis  Clementsport,  N.S.:  Canadian  Peacekeeping  Press,  2002. 

Eoust,  Joshua.  "Measuring  Success:  Are  We  Winning?",  May  2012. 

Gordon,  Michael,  and  Bernard  Trainor.  Cobra  li:  The  inside  Story  of  the  Invasion  and 
Occupation  of  Iraq.  New  York:  Vintage,  2006. 

Griffin,  Suzanne.  "Operational  Analysis  at  the  frontline  -  a  Generic  Approach  to  Measuring 
Progress?"  In  1 7ISMOR,  Session  J,  2000. 

Kagan,  Erederick  W.  "Measuring  Success."  Armed  Forces  Journal  (January  2006). 

Kapstem,  Ethan.  "Mihtary  Metrics:  How  Do  We  Know  When  We're  Winning  (or  Losing)  a 
War?".  Small  Wars  Journal  (July  2011). 

Kelly,  Justin,  and  David  Kilcullen.  "Chaos  Versus  Predictability:  A  Critique  of  Effects  Based 
Operations."  Australian  Army  Journal  2,  no.  1  (2006):  63-73. 

Kilcullen,  David.  "Measuring  Progress  in  Afghanistan."  In  Counterinsurgency  New  York: 
Oxford  University  Press,  2010. 

Kirhy,  Maurice  W.  Operational  Research  in  War  and  Peace:  The  British  Experience  from  the 
1930s  to  1970.  London:  Imperial  College  Press,  2003. 

Krepinevich,  Andrew  E.  "How  to  Win  in  Iraq."  Foreign  Affairs  84,  no.  5  (2005):  87-104. 

Lambert,  Nicholas  J.  "Measuring  the  Success  of  the  Nato  Operation  in  Bosnia  & 
Herzegovina  1995-2000."  In  17th  European  Conference  on  Operations  Research. 
Budapest,  Hungary,  2001. 

Managing  Military  Organizations:  Theory  and  Practice,  edited  by  Joseph  Soeters,  P.C.  van 
fenema  and  R.  Beeres  New  York:  Routledge,  2010. 

Marklund,  Joakrm,  Jermy  Lindoff,  Helene  Lackenbauer,  and  David  Harriman.  "Challenges 
in  Assessing  Progress  in  Multifunctional  Operations:  Experiences  from  a 
Provincial  Reconstruction  Team  in  Afghanistan."  In  16th  International  Command 
and  Control  Research  and  Technology  Symposium  Quebec,  Canada,  2011. 

Mattis,  James  N.  "U.S.  Joint  forces  Command  Commander's  Guidance  for  Effects-Based 
Operations."  Parameters  38,  no.  3  (Autumn  2008):  18-25. 

McChrystal,  Stanley.  "Comisaf's  Initial  Assessment."  2009. 


43 


CNA 


Military  Operations  Research  Society.  Assessments  of  Multinational  Operations:  From 
Analysis  to  Doctrine  and  Policy.  (Report  on  Proceedings  of  the  MORS  Special 
Meeting,  5-8  November  2012,  MacDill  AFB,  Tampa,  FL), 
http://www.mors.org/UserFiles/file/2012%20- 

%20Meetmg%20AMNO/MORS%20Assessments%20Special%20Meetrng%20Report%2 

0v4.pdf. 

Millett,  Allan  R.,  Williamson  Murray,  and  Kermeth  H.  Watman.  "The  Effectiveness  of 
Military  Organizations."  International  Security  11,  no.  1  (Summer  1986):  37-71. 

Morris,  John  C.,  and  Andrew  P.  Williams.  "The  Development  of  Theory-Driven  Evaluation 
in  the  Military."  American  Journal  of  Evaluation  30,  no.  1  (March  2009):  62-79. 

Neighbour,  M.R.,  P.  Bailey,  M.  Hawthorn,  C.  Eensing,  H.  Robson,  S.  Smith,  and  B. 
Zimmerman.  "Providing  Operational  Analysis  to  a  Peace  Support  Operation:  The 
Kosovo  Experience."  Journal  of  the  Operational  Research  Society  53,  no.  5  (2002): 
523-43. 

O'Hanlon,  Michael,  and  Hassina  Sherjan.  "Toughing  It  out  hr  Afghanistan."  2010. 

Owens,  Mackubin  T.  "The  Correlation  of  forces:  Then  and  Now."  Ashbrook  (February 
2004). 

Owens,  William  A.  "The  Emerging  System  of  Systems."  Military  Review  75,  no.  3  (1995). 

- .  "The  Once  and  Future  Revolution  in  Military  Affairs."  Joint  Forces  Quarterly,  no.  31 

(2002):  55-61. 

Porter,  Richard  E.  "Correlation  of  forces:  Revolutionary  Legacy."  Air  University  Review 
(March-April  1977). 

Puckett,  Timothy  R.  "Rule-Based  Expert  Systems  in  the  Command  Estimate:  An 
Operational  Perspective."  U.S.  Army  Command  and  General  Staff  College,  1990. 

Rosenzweig,  Phil.  "Mcnamara  and  the  Evolution  of  Modern  Business  Management." 
Harvard  Business  Review  (December  2010):  86-93. 

Schroden,  Jonathan.  "Measures  for  Security  in  a  Counterinsurgency."  Journal  of  Strategic 
Studies  32  (2009):  715-44. 

- .  "A  New  Paradigm  for  Assessment  in  Counterinsurgency."  Military  Operations 

Research  18,  no.  3  (2013):  5-20. 

- .  "Operations  Assessment  at  Isaf:  Changing  Paradigms."  In  Innovation  in  Operations 

Assessment:  Recent  Developments  in  Measuring  Results  in  Conflict  Environments, 
edited  by  Andrew  P.  Williams,  James  Bexfield,  Eabrizio  Eitzgerald  Farina  and 
Johannes  de  Nijs,  39-67.  Norfolk,  VA:  NATO  Supreme  Allied  Command 
Transformation,  2013. 

- .  "Why  Operations  Assessments  Fail:  It's  Not  Just  the  Metrics."  Naval  War  College 

Review  64,  no.  4  (Autumn  2011):  89-102. 


44 


CNA 


Shrader,  Charles  R.  History  of  Operations  Research  in  the  United  States  Army.  3  vols.  Vol. 
1,  Washington,  D.C.2006. 

- .  History  of  Operations  Research  in  the  United  States  Army.  3  vols.  Vol.  2, 

Washington,  D.C.  2008. 

Skinner,  Douglas  W.  Airland  Battle  Doctrine.  Cna  Professional  Paper  463.  1988. 

Smith,  Edward  A.  Effects  Based  Operations:  Applying  Network  Centric  Warfare  in  Peace, 
Crisis,  and  War.  Command  and  Control  Research  Program,  2002. 

Soucy,  Robert  R.,  Kevin  A.  Shwedo,  and  John  S.  Haven.  "War  Termination  Criteria  and 
Jopes:  A  Dangerous  Omission  in  Us  Crisis  Action  Planning."  Armed  Forces  Staff 
College,  1994. 

Thayer,  Thomas  C.  War  without  Fronts:  The  American  Experience  in  Vietnam  Boulder,  CO: 
Westview  Press,  1985. 

U.S.  Department  of  Defense.  1990  Joint  Military  Net  Assessment  Washington,  DC:  U.S. 
Department  of  Defense,  1990. 

- .  The  Implementation  of  Network-Centric  Warfare.  Washington,  DC:  U.S.  Department 

of  Defense,  2005. 

U.S.  Department  of  the  Army.  Operations.  Field  Manual  100-5.  Washington,  DC:  U.S. 
Department  of  the  Army,  1982. 

- .  The  Operations  Process.  Field  Manual  5-0.  Washington,  DC:  U.S.  Department  of  the 

Army,  2010. 

U.S.  Joint  Staff.  Commander’s  Handbook  for  Assessment  Planning  and  Execution 
Washington,  DC:  U.S.  Joint  Staff,  2011. 

- .  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms.  Joint 

Publication  1-02.  Washington,  DC:  U.S.  Joint  Staff,  2014. 

- .  Joint  Operation  Planning.  Joint  Publication  5-0.  Washington,  DC:  U.S.  Joint  Staff, 

2011. 

Upshur,  William,  Jonathan  Roginski,  and  David  Khcullen.  "Recognizing  Systems  in 
Afghanistan:  Fessons  Teamed  and  New  Approaches  to  Operational  Assessments." 
PRISM  3,  no.  3  (June  2012):  87-104. 

Vego,  Milan.  "Science  Vs.  The  Art  of  War."  Joint  Forces  Quarterly  66  (2012):  62-70. 

White  House.  "National  Security  Strategy."  2010. 

Womack,  James  K.  "Soviet  Correlation  of  Forces  and  Means:  Quantifying  Modern 
Operations."  US  Army  Command  and  Staff  College,  1990. 

Woodill,  Gerald,  Ewen  Stockbridge,  and  Mike  Cumberland.  "Defence  R&D  Canada  Centre 
for  Operational  Research  and  Analysis  Technical  Memorandum  2010-008."  In 


45 


CNA 


Counterinsurgency  Campaign  Assessment  and  Direction  in  Regional  Command 
South. 


46 


CNA 

ANALYSIS  &  SOLUTIONS 


CNA 

ANALYSIS  &  SOLUTIONS 


DOP-2014'U-006512- IRev 


CNA  Corporation  is  a  not-for-profit  research  organization 
that  serves  the  public  interest  by  providing 
in-depth  analysis  and  result-oriented  solutions 
to  help  government  leaders  choose 
the  best  course  of  action 
in  setting  policy  and  managing  operations. 


Nobody  gets  closer— 
to  the  people,  to  the  data,  to  the  problem. 


CNA 

ANALYSIS  &  SOLUTIONS 


www.cna.orq  •  703-824-2000 


3003  Washington  Boulevard,  Arlington,  VA  22201 


